Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 706770 Details for
Bug 918437
timestamps updated after self-heal following primary brick rebuild
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
self-heal trace log
glustershd.log (text/plain), 273.33 KB, created by
Todd Stansell
on 2013-03-07 19:12:44 UTC
(
hide
)
Description:
self-heal trace log
Filename:
MIME Type:
Creator:
Todd Stansell
Created:
2013-03-07 19:12:44 UTC
Size:
273.33 KB
patch
obsolete
>[2013-03-07 17:00:37.459049] E [afr-common.c:3761:afr_notify] 0-gv3-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2013-03-07 17:00:39.237462] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect >[2013-03-07 17:00:39.237514] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:39.237535] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007 >[2013-03-07 17:00:39.237550] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:39.237562] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:39.243415] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:39.243457] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 7 >[2013-03-07 17:00:39.243476] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 7, interval 2, idle: 20 >[2013-03-07 17:00:39.243510] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:39.243582] E [socket.c:2767:socket_connect] 0-gv3-client-0: connection attempt failed (Connection refused) >[2013-03-07 17:00:42.243906] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect >[2013-03-07 17:00:42.243956] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:42.243976] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007 >[2013-03-07 17:00:42.243992] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:42.244005] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:42.249988] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:42.250039] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 7 >[2013-03-07 17:00:42.250058] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 7, interval 2, idle: 20 >[2013-03-07 17:00:42.250095] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:42.250168] E [socket.c:2767:socket_connect] 0-gv3-client-0: connection attempt failed (Connection refused) >[2013-03-07 17:00:45.250523] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect >[2013-03-07 17:00:45.255828] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-127.0.0.1 (port-24007) for hostname: localhost and port: 24007 >[2013-03-07 17:00:45.255872] T [socket.c:731:__socket_nodelay] 0-glusterfs: NODELAY enabled for socket 7 >[2013-03-07 17:00:45.255891] T [socket.c:790:__socket_keepalive] 0-glusterfs: Keep-alive enabled for socket 7, interval 2, idle: 20 >[2013-03-07 17:00:45.255928] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:45.256036] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect >[2013-03-07 17:00:45.256073] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:45.256080] T [rpc-clnt.c:1301:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:45.256101] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007 >[2013-03-07 17:00:45.256120] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 32, rpc hdr: 64 >[2013-03-07 17:00:45.256125] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:45.256168] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:45.256204] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 2) to rpc-transport (glusterfs) >[2013-03-07 17:00:45.256244] D [glusterfsd-mgmt.c:2152:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given >[2013-03-07 17:00:45.262216] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:45.262269] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 8 >[2013-03-07 17:00:45.262290] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 8, interval 2, idle: 20 >[2013-03-07 17:00:45.262325] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:45.262469] D [client.c:2050:client_rpc_notify] 0-gv3-client-0: got RPC_CLNT_CONNECT >[2013-03-07 17:00:45.262506] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:45.262527] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:45.262596] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7f43f77e8f42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7f43f77cc0c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected >[2013-03-07 17:00:45.262662] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x93x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:45.262689] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:45.611442] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 2) from rpc-transport (glusterfs) >[2013-03-07 17:00:45.611500] I [glusterfsd-mgmt.c:1583:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2013-03-07 17:00:45.611539] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x93x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:45.611566] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-0: detected portmapper on server >[2013-03-07 17:00:45.611594] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:45.611613] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64 >[2013-03-07 17:00:45.611688] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7f43f77e7762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7f43f77cc0c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected >[2013-03-07 17:00:45.611749] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x94x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:45.611774] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:45.611876] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x94x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:45.611920] D [client-handshake.c:1741:client_query_portmap_cbk] 0-gv3-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2013-03-07 17:00:45.611978] D [socket.c:486:__socket_rwv] 0-gv3-client-0: EOF on socket >[2013-03-07 17:00:45.612001] W [socket.c:514:__socket_rwv] 0-gv3-client-0: readv failed (No data available) >[2013-03-07 17:00:45.612016] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-0: reading from socket failed. Error (No data available), peer (172.22.6.136:24007) >[2013-03-07 17:00:45.612031] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:45.612059] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-0: cleaning up state in transport object 0xa4e010 >[2013-03-07 17:00:45.612079] I [client.c:2097:client_rpc_notify] 0-gv3-client-0: disconnected >[2013-03-07 17:00:45.857663] W [glusterfsd.c:970:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x398f6e767d] (-->/lib64/libpthread.so.0() [0x398fa07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x40523d]))) 0-: received signum (15), shutting down >[2013-03-07 17:00:45.857691] D [glusterfsd-mgmt.c:2214:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given >[2013-03-07 17:00:45.857721] T [glusterfsd.c:1489:glusterfs_pidfile_cleanup] 0-glusterfsd: pidfile /var/lib/glusterd/glustershd/run/glustershd.pid cleanup >[2013-03-07 17:00:46.870036] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0alpha (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/876d8b0cbe47b5e0a51969bcdebaac27.socket --xlator-option *replicate*.node-uuid=860e1ecd-78c7-4a99-a642-e8eac0919a56) >[2013-03-07 17:00:46.873936] I [socket.c:3459:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled >[2013-03-07 17:00:46.873995] I [socket.c:3474:socket_init] 0-socket.glusterfsd: using system polling thread >[2013-03-07 17:00:46.874182] I [socket.c:3459:socket_init] 0-glusterfs: SSL support is NOT enabled >[2013-03-07 17:00:46.874216] I [socket.c:3474:socket_init] 0-glusterfs: using system polling thread >[2013-03-07 17:00:46.884836] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.886417] I [graph.c:239:gf_add_cmdline_options] 0-gv13-replicate-0: adding option 'node-uuid' for volume 'gv13-replicate-0' with value '860e1ecd-78c7-4a99-a642-e8eac0919a56' >[2013-03-07 17:00:46.886429] I [graph.c:239:gf_add_cmdline_options] 0-gv3-replicate-0: adding option 'node-uuid' for volume 'gv3-replicate-0' with value '860e1ecd-78c7-4a99-a642-e8eac0919a56' >[2013-03-07 17:00:46.886437] I [graph.c:239:gf_add_cmdline_options] 0-gv14-replicate-0: adding option 'node-uuid' for volume 'gv14-replicate-0' with value '860e1ecd-78c7-4a99-a642-e8eac0919a56' >[2013-03-07 17:00:46.886492] T [options.c:1111:xlator_option_init_xlator] 0-gv13-replicate-0: option read-subvolume not set >[2013-03-07 17:00:46.886508] T [options.c:1106:xlator_option_init_int32] 0-gv13-replicate-0: option read-subvolume-index using default value -1 >[2013-03-07 17:00:46.886515] T [options.c:77:xlator_option_validate_int] 0-gv13-replicate-0: no range check required for 'option read-subvolume-index -1' >[2013-03-07 17:00:46.886521] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option choose-local using default value true >[2013-03-07 17:00:46.886528] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option read-hash-mode using default value 0 >[2013-03-07 17:00:46.886535] T [options.c:1111:xlator_option_init_xlator] 0-gv13-replicate-0: option favorite-child not set >[2013-03-07 17:00:46.886542] D [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option background-self-heal-count using set value 0 >[2013-03-07 17:00:46.886548] D [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option data-self-heal using set value on >[2013-03-07 17:00:46.886555] T [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option data-self-heal-algorithm not set >[2013-03-07 17:00:46.886561] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option data-self-heal-window-size using default value 1 >[2013-03-07 17:00:46.886567] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option metadata-self-heal using set value on >[2013-03-07 17:00:46.886573] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option entry-self-heal using set value on >[2013-03-07 17:00:46.886579] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option self-heal-daemon using set value on >[2013-03-07 17:00:46.886586] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option iam-self-heal-daemon using set value yes >[2013-03-07 17:00:46.886592] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option data-change-log using default value on >[2013-03-07 17:00:46.886599] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option metadata-change-log using default value on >[2013-03-07 17:00:46.886605] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option entry-change-log using default value on >[2013-03-07 17:00:46.886611] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option optimistic-change-log using default value on >[2013-03-07 17:00:46.886617] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option inodelk-trace using default value off >[2013-03-07 17:00:46.886624] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option entrylk-trace using default value off >[2013-03-07 17:00:46.886630] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option strict-readdir using default value off >[2013-03-07 17:00:46.886636] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option eager-lock using default value on >[2013-03-07 17:00:46.886642] T [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option quorum-type using default value none >[2013-03-07 17:00:46.886649] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option quorum-count not set >[2013-03-07 17:00:46.886660] T [options.c:1107:xlator_option_init_size] 0-gv13-replicate-0: option self-heal-readdir-size using default value 1KB >[2013-03-07 17:00:46.886674] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option post-op-delay-secs using default value 1 >[2013-03-07 17:00:46.886681] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option readdir-failover using default value on >[2013-03-07 17:00:46.888320] D [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option node-uuid using set value 860e1ecd-78c7-4a99-a642-e8eac0919a56 >[2013-03-07 17:00:46.888332] T [options.c:1106:xlator_option_init_int32] 0-gv13-replicate-0: option heal-timeout using default value 600 >[2013-03-07 17:00:46.888341] D [client.c:2331:client_init_grace_timer] 0-gv13-client-1: lk-heal = off >[2013-03-07 17:00:46.888348] D [client.c:2342:client_init_grace_timer] 0-gv13-client-1: Client grace timeout value = 10 >[2013-03-07 17:00:46.888354] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-1: option frame-timeout using default value 1800 >[2013-03-07 17:00:46.888361] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-1: option remote-port not set >[2013-03-07 17:00:46.888367] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-1: option ping-timeout using default value 42 >[2013-03-07 17:00:46.888373] D [options.c:1112:xlator_option_init_path] 0-gv13-client-1: option remote-subvolume using set value /gfs0/brick13 >[2013-03-07 17:00:46.888380] T [options.c:1110:xlator_option_init_bool] 0-gv13-client-1: option filter-O_DIRECT using default value disable >[2013-03-07 17:00:46.888904] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv13-client-1: defaulting frame-timeout to 30mins >[2013-03-07 17:00:46.888917] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so >[2013-03-07 17:00:46.888935] I [socket.c:3459:socket_init] 0-gv13-client-1: SSL support is NOT enabled >[2013-03-07 17:00:46.888942] I [socket.c:3474:socket_init] 0-gv13-client-1: using system polling thread >[2013-03-07 17:00:46.888948] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv13-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-03-07 17:00:46.888957] D [client.c:2306:client_init_rpc] 0-gv13-client-1: client init successful >[2013-03-07 17:00:46.888963] D [client.c:2331:client_init_grace_timer] 0-gv13-client-0: lk-heal = off >[2013-03-07 17:00:46.888968] D [client.c:2342:client_init_grace_timer] 0-gv13-client-0: Client grace timeout value = 10 >[2013-03-07 17:00:46.888974] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-0: option frame-timeout using default value 1800 >[2013-03-07 17:00:46.888981] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-0: option remote-port not set >[2013-03-07 17:00:46.888986] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-0: option ping-timeout using default value 42 >[2013-03-07 17:00:46.888992] D [options.c:1112:xlator_option_init_path] 0-gv13-client-0: option remote-subvolume using set value /gfs0/brick13 >[2013-03-07 17:00:46.888999] T [options.c:1110:xlator_option_init_bool] 0-gv13-client-0: option filter-O_DIRECT using default value disable >[2013-03-07 17:00:46.889539] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv13-client-0: defaulting frame-timeout to 30mins >[2013-03-07 17:00:46.889551] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so >[2013-03-07 17:00:46.889567] I [socket.c:3459:socket_init] 0-gv13-client-0: SSL support is NOT enabled >[2013-03-07 17:00:46.889573] I [socket.c:3474:socket_init] 0-gv13-client-0: using system polling thread >[2013-03-07 17:00:46.889579] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv13-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-03-07 17:00:46.889585] D [client.c:2306:client_init_rpc] 0-gv13-client-0: client init successful >[2013-03-07 17:00:46.889596] T [options.c:1111:xlator_option_init_xlator] 0-gv3-replicate-0: option read-subvolume not set >[2013-03-07 17:00:46.889603] T [options.c:1106:xlator_option_init_int32] 0-gv3-replicate-0: option read-subvolume-index using default value -1 >[2013-03-07 17:00:46.889609] T [options.c:77:xlator_option_validate_int] 0-gv3-replicate-0: no range check required for 'option read-subvolume-index -1' >[2013-03-07 17:00:46.889615] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option choose-local using default value true >[2013-03-07 17:00:46.889621] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option read-hash-mode using default value 0 >[2013-03-07 17:00:46.889628] T [options.c:1111:xlator_option_init_xlator] 0-gv3-replicate-0: option favorite-child not set >[2013-03-07 17:00:46.889634] D [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option background-self-heal-count using set value 0 >[2013-03-07 17:00:46.889640] D [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option data-self-heal using set value on >[2013-03-07 17:00:46.889647] T [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option data-self-heal-algorithm not set >[2013-03-07 17:00:46.889653] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option data-self-heal-window-size using default value 1 >[2013-03-07 17:00:46.889660] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option metadata-self-heal using set value on >[2013-03-07 17:00:46.889666] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option entry-self-heal using set value on >[2013-03-07 17:00:46.889672] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option self-heal-daemon using set value on >[2013-03-07 17:00:46.889678] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option iam-self-heal-daemon using set value yes >[2013-03-07 17:00:46.889685] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option data-change-log using default value on >[2013-03-07 17:00:46.889691] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option metadata-change-log using default value on >[2013-03-07 17:00:46.889697] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option entry-change-log using default value on >[2013-03-07 17:00:46.889703] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option optimistic-change-log using default value on >[2013-03-07 17:00:46.889709] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option inodelk-trace using default value off >[2013-03-07 17:00:46.889716] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option entrylk-trace using default value off >[2013-03-07 17:00:46.889722] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option strict-readdir using default value off >[2013-03-07 17:00:46.889728] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option eager-lock using default value on >[2013-03-07 17:00:46.889734] T [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option quorum-type using default value none >[2013-03-07 17:00:46.889741] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option quorum-count not set >[2013-03-07 17:00:46.889747] T [options.c:1107:xlator_option_init_size] 0-gv3-replicate-0: option self-heal-readdir-size using default value 1KB >[2013-03-07 17:00:46.889755] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option post-op-delay-secs using default value 1 >[2013-03-07 17:00:46.889762] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option readdir-failover using default value on >[2013-03-07 17:00:46.891526] D [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option node-uuid using set value 860e1ecd-78c7-4a99-a642-e8eac0919a56 >[2013-03-07 17:00:46.891540] T [options.c:1106:xlator_option_init_int32] 0-gv3-replicate-0: option heal-timeout using default value 600 >[2013-03-07 17:00:46.891548] D [client.c:2331:client_init_grace_timer] 0-gv3-client-2: lk-heal = off >[2013-03-07 17:00:46.891557] D [client.c:2342:client_init_grace_timer] 0-gv3-client-2: Client grace timeout value = 10 >[2013-03-07 17:00:46.891564] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-2: option frame-timeout using default value 1800 >[2013-03-07 17:00:46.891570] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-2: option remote-port not set >[2013-03-07 17:00:46.891576] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-2: option ping-timeout using default value 42 >[2013-03-07 17:00:46.891582] D [options.c:1112:xlator_option_init_path] 0-gv3-client-2: option remote-subvolume using set value /gfs0/brick3 >[2013-03-07 17:00:46.891589] T [options.c:1110:xlator_option_init_bool] 0-gv3-client-2: option filter-O_DIRECT using default value disable >[2013-03-07 17:00:46.892133] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv3-client-2: defaulting frame-timeout to 30mins >[2013-03-07 17:00:46.892144] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so >[2013-03-07 17:00:46.892161] I [socket.c:3459:socket_init] 0-gv3-client-2: SSL support is NOT enabled >[2013-03-07 17:00:46.892167] I [socket.c:3474:socket_init] 0-gv3-client-2: using system polling thread >[2013-03-07 17:00:46.892173] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv3-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-03-07 17:00:46.892179] D [client.c:2306:client_init_rpc] 0-gv3-client-2: client init successful >[2013-03-07 17:00:46.892185] D [client.c:2331:client_init_grace_timer] 0-gv3-client-1: lk-heal = off >[2013-03-07 17:00:46.892190] D [client.c:2342:client_init_grace_timer] 0-gv3-client-1: Client grace timeout value = 10 >[2013-03-07 17:00:46.892196] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-1: option frame-timeout using default value 1800 >[2013-03-07 17:00:46.892202] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-1: option remote-port not set >[2013-03-07 17:00:46.892208] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-1: option ping-timeout using default value 42 >[2013-03-07 17:00:46.892214] D [options.c:1112:xlator_option_init_path] 0-gv3-client-1: option remote-subvolume using set value /gfs0/brick2 >[2013-03-07 17:00:46.892221] T [options.c:1110:xlator_option_init_bool] 0-gv3-client-1: option filter-O_DIRECT using default value disable >[2013-03-07 17:00:46.892760] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv3-client-1: defaulting frame-timeout to 30mins >[2013-03-07 17:00:46.892771] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so >[2013-03-07 17:00:46.892786] I [socket.c:3459:socket_init] 0-gv3-client-1: SSL support is NOT enabled >[2013-03-07 17:00:46.892795] I [socket.c:3474:socket_init] 0-gv3-client-1: using system polling thread >[2013-03-07 17:00:46.892801] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv3-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-03-07 17:00:46.892807] D [client.c:2306:client_init_rpc] 0-gv3-client-1: client init successful >[2013-03-07 17:00:46.892813] D [client.c:2331:client_init_grace_timer] 0-gv3-client-0: lk-heal = off >[2013-03-07 17:00:46.892818] D [client.c:2342:client_init_grace_timer] 0-gv3-client-0: Client grace timeout value = 10 >[2013-03-07 17:00:46.892823] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-0: option frame-timeout using default value 1800 >[2013-03-07 17:00:46.892830] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-0: option remote-port not set >[2013-03-07 17:00:46.892835] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-0: option ping-timeout using default value 42 >[2013-03-07 17:00:46.892842] D [options.c:1112:xlator_option_init_path] 0-gv3-client-0: option remote-subvolume using set value /gfs0/brick1 >[2013-03-07 17:00:46.892848] T [options.c:1110:xlator_option_init_bool] 0-gv3-client-0: option filter-O_DIRECT using default value disable >[2013-03-07 17:00:46.893363] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv3-client-0: defaulting frame-timeout to 30mins >[2013-03-07 17:00:46.893377] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so >[2013-03-07 17:00:46.893392] I [socket.c:3459:socket_init] 0-gv3-client-0: SSL support is NOT enabled >[2013-03-07 17:00:46.893398] I [socket.c:3474:socket_init] 0-gv3-client-0: using system polling thread >[2013-03-07 17:00:46.893404] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv3-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-03-07 17:00:46.893410] D [client.c:2306:client_init_rpc] 0-gv3-client-0: client init successful >[2013-03-07 17:00:46.893417] T [options.c:1111:xlator_option_init_xlator] 0-gv14-replicate-0: option read-subvolume not set >[2013-03-07 17:00:46.893423] T [options.c:1106:xlator_option_init_int32] 0-gv14-replicate-0: option read-subvolume-index using default value -1 >[2013-03-07 17:00:46.893429] T [options.c:77:xlator_option_validate_int] 0-gv14-replicate-0: no range check required for 'option read-subvolume-index -1' >[2013-03-07 17:00:46.893435] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option choose-local using default value true >[2013-03-07 17:00:46.893441] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option read-hash-mode using default value 0 >[2013-03-07 17:00:46.893448] T [options.c:1111:xlator_option_init_xlator] 0-gv14-replicate-0: option favorite-child not set >[2013-03-07 17:00:46.893454] D [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option background-self-heal-count using set value 0 >[2013-03-07 17:00:46.893460] D [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option data-self-heal using set value on >[2013-03-07 17:00:46.893467] T [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option data-self-heal-algorithm not set >[2013-03-07 17:00:46.893473] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option data-self-heal-window-size using default value 1 >[2013-03-07 17:00:46.893480] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option metadata-self-heal using set value on >[2013-03-07 17:00:46.893486] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option entry-self-heal using set value on >[2013-03-07 17:00:46.893493] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option self-heal-daemon using set value on >[2013-03-07 17:00:46.893499] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option iam-self-heal-daemon using set value yes >[2013-03-07 17:00:46.893517] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option data-change-log using default value on >[2013-03-07 17:00:46.893524] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option metadata-change-log using default value on >[2013-03-07 17:00:46.893530] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option entry-change-log using default value on >[2013-03-07 17:00:46.893537] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option optimistic-change-log using default value on >[2013-03-07 17:00:46.893543] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option inodelk-trace using default value off >[2013-03-07 17:00:46.893549] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option entrylk-trace using default value off >[2013-03-07 17:00:46.893556] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option strict-readdir using default value off >[2013-03-07 17:00:46.893562] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option eager-lock using default value on >[2013-03-07 17:00:46.893568] T [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option quorum-type using default value none >[2013-03-07 17:00:46.893575] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option quorum-count not set >[2013-03-07 17:00:46.893581] T [options.c:1107:xlator_option_init_size] 0-gv14-replicate-0: option self-heal-readdir-size using default value 1KB >[2013-03-07 17:00:46.893594] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option post-op-delay-secs using default value 1 >[2013-03-07 17:00:46.893602] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option readdir-failover using default value on >[2013-03-07 17:00:46.895341] D [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option node-uuid using set value 860e1ecd-78c7-4a99-a642-e8eac0919a56 >[2013-03-07 17:00:46.895353] T [options.c:1106:xlator_option_init_int32] 0-gv14-replicate-0: option heal-timeout using default value 600 >[2013-03-07 17:00:46.895361] D [client.c:2331:client_init_grace_timer] 0-gv14-client-1: lk-heal = off >[2013-03-07 17:00:46.895366] D [client.c:2342:client_init_grace_timer] 0-gv14-client-1: Client grace timeout value = 10 >[2013-03-07 17:00:46.895372] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-1: option frame-timeout using default value 1800 >[2013-03-07 17:00:46.895378] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-1: option remote-port not set >[2013-03-07 17:00:46.895385] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-1: option ping-timeout using default value 42 >[2013-03-07 17:00:46.895391] D [options.c:1112:xlator_option_init_path] 0-gv14-client-1: option remote-subvolume using set value /gfs0/brick14 >[2013-03-07 17:00:46.895397] T [options.c:1110:xlator_option_init_bool] 0-gv14-client-1: option filter-O_DIRECT using default value disable >[2013-03-07 17:00:46.895931] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv14-client-1: defaulting frame-timeout to 30mins >[2013-03-07 17:00:46.895943] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so >[2013-03-07 17:00:46.895959] I [socket.c:3459:socket_init] 0-gv14-client-1: SSL support is NOT enabled >[2013-03-07 17:00:46.895966] I [socket.c:3474:socket_init] 0-gv14-client-1: using system polling thread >[2013-03-07 17:00:46.895972] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv14-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-03-07 17:00:46.895977] D [client.c:2306:client_init_rpc] 0-gv14-client-1: client init successful >[2013-03-07 17:00:46.895983] D [client.c:2331:client_init_grace_timer] 0-gv14-client-0: lk-heal = off >[2013-03-07 17:00:46.895989] D [client.c:2342:client_init_grace_timer] 0-gv14-client-0: Client grace timeout value = 10 >[2013-03-07 17:00:46.895994] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-0: option frame-timeout using default value 1800 >[2013-03-07 17:00:46.896001] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-0: option remote-port not set >[2013-03-07 17:00:46.896006] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-0: option ping-timeout using default value 42 >[2013-03-07 17:00:46.896013] D [options.c:1112:xlator_option_init_path] 0-gv14-client-0: option remote-subvolume using set value /gfs0/brick14 >[2013-03-07 17:00:46.896019] T [options.c:1110:xlator_option_init_bool] 0-gv14-client-0: option filter-O_DIRECT using default value disable >[2013-03-07 17:00:46.896551] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv14-client-0: defaulting frame-timeout to 30mins >[2013-03-07 17:00:46.896563] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so >[2013-03-07 17:00:46.896580] I [socket.c:3459:socket_init] 0-gv14-client-0: SSL support is NOT enabled >[2013-03-07 17:00:46.896587] I [socket.c:3474:socket_init] 0-gv14-client-0: using system polling thread >[2013-03-07 17:00:46.896593] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv14-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 >[2013-03-07 17:00:46.896598] D [client.c:2306:client_init_rpc] 0-gv14-client-0: client init successful >[2013-03-07 17:00:46.896625] I [client.c:2154:notify] 0-gv13-client-0: parent translators are ready, attempting connect on transport >[2013-03-07 17:00:46.896631] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-0: attempting reconnect >[2013-03-07 17:00:46.896643] D [name.c:155:client_fill_address_family] 0-gv13-client-0: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.896649] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-0: option remote-port missing in volume gv13-client-0. Defaulting to 24007 >[2013-03-07 17:00:46.896656] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.902895] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.902955] T [socket.c:731:__socket_nodelay] 0-gv13-client-0: NODELAY enabled for socket 10 >[2013-03-07 17:00:46.902976] T [socket.c:790:__socket_keepalive] 0-gv13-client-0: Keep-alive enabled for socket 10, interval 2, idle: 20 >[2013-03-07 17:00:46.903023] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.903170] I [client.c:2154:notify] 0-gv13-client-1: parent translators are ready, attempting connect on transport >[2013-03-07 17:00:46.903195] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-1: attempting reconnect >[2013-03-07 17:00:46.903214] D [name.c:155:client_fill_address_family] 0-gv13-client-1: address-family not specified, guessing it to be inet from (remote-host: host14) >[2013-03-07 17:00:46.903229] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-1: option remote-port missing in volume gv13-client-1. Defaulting to 24007 >[2013-03-07 17:00:46.903244] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14 >[2013-03-07 17:00:46.925853] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007 >[2013-03-07 17:00:46.925869] T [socket.c:731:__socket_nodelay] 0-gv13-client-1: NODELAY enabled for socket 11 >[2013-03-07 17:00:46.925875] T [socket.c:790:__socket_keepalive] 0-gv13-client-1: Keep-alive enabled for socket 11, interval 2, idle: 20 >[2013-03-07 17:00:46.925887] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.926099] I [client.c:2154:notify] 0-gv3-client-0: parent translators are ready, attempting connect on transport >[2013-03-07 17:00:46.926108] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect >[2013-03-07 17:00:46.926115] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.926120] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007 >[2013-03-07 17:00:46.926124] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.928253] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.928266] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 12 >[2013-03-07 17:00:46.928272] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 12, interval 2, idle: 20 >[2013-03-07 17:00:46.928282] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.928316] I [client.c:2154:notify] 0-gv3-client-1: parent translators are ready, attempting connect on transport >[2013-03-07 17:00:46.928337] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-1: attempting reconnect >[2013-03-07 17:00:46.928344] D [name.c:155:client_fill_address_family] 0-gv3-client-1: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.928349] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-1: option remote-port missing in volume gv3-client-1. Defaulting to 24007 >[2013-03-07 17:00:46.928359] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.930513] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.930530] T [socket.c:731:__socket_nodelay] 0-gv3-client-1: NODELAY enabled for socket 13 >[2013-03-07 17:00:46.930536] T [socket.c:790:__socket_keepalive] 0-gv3-client-1: Keep-alive enabled for socket 13, interval 2, idle: 20 >[2013-03-07 17:00:46.930547] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.930583] I [client.c:2154:notify] 0-gv3-client-2: parent translators are ready, attempting connect on transport >[2013-03-07 17:00:46.930603] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-2: attempting reconnect >[2013-03-07 17:00:46.930610] D [name.c:155:client_fill_address_family] 0-gv3-client-2: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.930615] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-2: option remote-port missing in volume gv3-client-2. Defaulting to 24007 >[2013-03-07 17:00:46.930620] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.932807] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.932820] T [socket.c:731:__socket_nodelay] 0-gv3-client-2: NODELAY enabled for socket 14 >[2013-03-07 17:00:46.932826] T [socket.c:790:__socket_keepalive] 0-gv3-client-2: Keep-alive enabled for socket 14, interval 2, idle: 20 >[2013-03-07 17:00:46.932836] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.932883] I [client.c:2154:notify] 0-gv14-client-0: parent translators are ready, attempting connect on transport >[2013-03-07 17:00:46.932892] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-0: attempting reconnect >[2013-03-07 17:00:46.932899] D [name.c:155:client_fill_address_family] 0-gv14-client-0: address-family not specified, guessing it to be inet from (remote-host: host14) >[2013-03-07 17:00:46.932904] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-0: option remote-port missing in volume gv14-client-0. Defaulting to 24007 >[2013-03-07 17:00:46.932908] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14 >[2013-03-07 17:00:46.935058] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007 >[2013-03-07 17:00:46.935071] T [socket.c:731:__socket_nodelay] 0-gv14-client-0: NODELAY enabled for socket 15 >[2013-03-07 17:00:46.935076] T [socket.c:790:__socket_keepalive] 0-gv14-client-0: Keep-alive enabled for socket 15, interval 2, idle: 20 >[2013-03-07 17:00:46.935087] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.935372] I [client.c:2154:notify] 0-gv14-client-1: parent translators are ready, attempting connect on transport >[2013-03-07 17:00:46.935421] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-1: attempting reconnect >[2013-03-07 17:00:46.935442] D [name.c:155:client_fill_address_family] 0-gv14-client-1: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.935458] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-1: option remote-port missing in volume gv14-client-1. Defaulting to 24007 >[2013-03-07 17:00:46.935473] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.937880] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.937900] T [socket.c:731:__socket_nodelay] 0-gv14-client-1: NODELAY enabled for socket 16 >[2013-03-07 17:00:46.937911] T [socket.c:790:__socket_keepalive] 0-gv14-client-1: Keep-alive enabled for socket 16, interval 2, idle: 20 >[2013-03-07 17:00:46.937924] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume gv14-client-0 > 2: type protocol/client > 3: option password f2c735fd-01d5-416c-b7cc-643565a19e30 > 4: option username 98840dc8-2d17-4c79-b9a0-62700d79daa9 > 5: option transport-type tcp > 6: option remote-subvolume /gfs0/brick14 > 7: option remote-host host14 > 8: end-volume > 9: > 10: volume gv14-client-1 > 11: type protocol/client > 12: option password f2c735fd-01d5-416c-b7cc-643565a19e30 > 13: option username 98840dc8-2d17-4c79-b9a0-62700d79daa9 > 14: option transport-type tcp > 15: option remote-subvolume /gfs0/brick14 > 16: option remote-host host13 > 17: end-volume > 18: > 19: volume gv14-replicate-0 > 20: type cluster/replicate > 21: option iam-self-heal-daemon yes > 22: option self-heal-daemon on > 23: option entry-self-heal on > 24: option data-self-heal on > 25: option metadata-self-heal on > 26: option background-self-heal-count 0 > 27: subvolumes gv14-client-0 gv14-client-1 > 28: end-volume > 29: > 30: volume gv3-client-0 > 31: type protocol/client > 32: option password 85e393d6-2d14-4c47-801f-0d01436dc457 > 33: option username f29c81ad-73cb-49b4-a0ea-2dcfda202b4b > 34: option transport-type tcp > 35: option remote-subvolume /gfs0/brick1 > 36: option remote-host host13 > 37: end-volume > 38: > 39: volume gv3-client-1 > 40: type protocol/client > 41: option password 85e393d6-2d14-4c47-801f-0d01436dc457 > 42: option username f29c81ad-73cb-49b4-a0ea-2dcfda202b4b > 43: option transport-type tcp > 44: option remote-subvolume /gfs0/brick2 > 45: option remote-host host13 > 46: end-volume > 47: > 48: volume gv3-client-2 > 49: type protocol/client > 50: option password 85e393d6-2d14-4c47-801f-0d01436dc457 > 51: option username f29c81ad-73cb-49b4-a0ea-2dcfda202b4b > 52: option transport-type tcp > 53: option remote-subvolume /gfs0/brick3 > 54: option remote-host host13 > 55: end-volume > 56: > 57: volume gv3-replicate-0 > 58: type cluster/replicate > 59: option iam-self-heal-daemon yes > 60: option self-heal-daemon on > 61: option entry-self-heal on > 62: option data-self-heal on > 63: option metadata-self-heal on > 64: option background-self-heal-count 0 > 65: subvolumes gv3-client-0 gv3-client-1 gv3-client-2 > 66: end-volume > 67: > 68: volume gv13-client-0 > 69: type protocol/client > 70: option password a5bb356d-e400-47b5-99da-dc91b78929b1 > 71: option username 305f0d14-54f0-4d35-991c-e76d89f4b307 > 72: option transport-type tcp > 73: option remote-subvolume /gfs0/brick13 > 74: option remote-host host13 > 75: end-volume > 76: > 77: volume gv13-client-1 > 78: type protocol/client > 79: option password a5bb356d-e400-47b5-99da-dc91b78929b1 > 80: option username 305f0d14-54f0-4d35-991c-e76d89f4b307 > 81: option transport-type tcp > 82: option remote-subvolume /gfs0/brick13 > 83: option remote-host host14 > 84: end-volume > 85: > 86: volume gv13-replicate-0 > 87: type cluster/replicate > 88: option iam-self-heal-daemon yes > 89: option self-heal-daemon on > 90: option entry-self-heal on > 91: option data-self-heal on > 92: option metadata-self-heal on > 93: option background-self-heal-count 0 > 94: subvolumes gv13-client-0 gv13-client-1 > 95: end-volume > 96: > 97: volume glustershd > 98: type debug/io-stats > 99: option log-level TRACE >100: subvolumes gv13-replicate-0 gv3-replicate-0 gv14-replicate-0 >101: end-volume > >+------------------------------------------------------------------------------+ >[2013-03-07 17:00:46.938045] D [glusterfsd-mgmt.c:2152:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given >[2013-03-07 17:00:46.938077] D [client.c:2050:client_rpc_notify] 0-gv13-client-0: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.938104] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938115] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.938265] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.938289] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.938302] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938315] D [client.c:2050:client_rpc_notify] 0-gv13-client-1: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.938323] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938329] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.938347] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.938360] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.938367] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938377] D [client.c:2050:client_rpc_notify] 0-gv3-client-0: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.938391] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938408] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.938425] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.938441] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.938448] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938469] D [client.c:2050:client_rpc_notify] 0-gv3-client-1: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.938477] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938482] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.938498] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.938518] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.938529] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938539] D [client.c:2050:client_rpc_notify] 0-gv3-client-2: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.938547] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938553] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.938568] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected >[2013-03-07 17:00:46.938594] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.938608] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938618] D [client.c:2050:client_rpc_notify] 0-gv14-client-0: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.938624] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938633] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.938660] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.938672] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.938678] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938688] D [client.c:2050:client_rpc_notify] 0-gv14-client-1: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.938697] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938702] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.938717] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.938743] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.938751] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938768] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.938780] D [client-handshake.c:1692:server_has_portmap] 0-gv13-client-0: detected portmapper on server >[2013-03-07 17:00:46.938790] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938810] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64 >[2013-03-07 17:00:46.938829] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.938841] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.938848] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938860] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.938868] D [client-handshake.c:1692:server_has_portmap] 0-gv13-client-1: detected portmapper on server >[2013-03-07 17:00:46.938876] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938881] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64 >[2013-03-07 17:00:46.938897] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.938918] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.938925] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.938935] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.938941] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-0: detected portmapper on server >[2013-03-07 17:00:46.938949] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.938954] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64 >[2013-03-07 17:00:46.938968] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.938980] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.938998] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.939009] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.939016] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-1: detected portmapper on server >[2013-03-07 17:00:46.939023] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.939032] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64 >[2013-03-07 17:00:46.939047] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.939060] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.939077] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.939087] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.939094] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-2: detected portmapper on server >[2013-03-07 17:00:46.939101] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.939105] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64 >[2013-03-07 17:00:46.939119] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected >[2013-03-07 17:00:46.939149] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.939156] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.939167] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.939174] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv13-client-0: changing port to 49152 (from 0) >[2013-03-07 17:00:46.939190] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.939196] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv13-client-1: changing port to 49152 (from 0) >[2013-03-07 17:00:46.939206] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.939212] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv3-client-0: changing port to 49154 (from 0) >[2013-03-07 17:00:46.939226] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.939234] D [client-handshake.c:1692:server_has_portmap] 0-gv14-client-1: detected portmapper on server >[2013-03-07 17:00:46.939242] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.939247] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64 >[2013-03-07 17:00:46.939262] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.939274] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.939284] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.939295] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.939302] D [client-handshake.c:1692:server_has_portmap] 0-gv14-client-0: detected portmapper on server >[2013-03-07 17:00:46.939309] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.939315] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64 >[2013-03-07 17:00:46.939341] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.939351] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.939357] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.939365] D [socket.c:486:__socket_rwv] 0-gv13-client-0: EOF on socket >[2013-03-07 17:00:46.939372] W [socket.c:514:__socket_rwv] 0-gv13-client-0: readv failed (No data available) >[2013-03-07 17:00:46.939377] D [socket.c:1962:__socket_proto_state_machine] 0-gv13-client-0: reading from socket failed. Error (No data available), peer (172.22.6.136:24007) >[2013-03-07 17:00:46.939405] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:46.939416] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv13-client-0: cleaning up state in transport object 0x9d2140 >[2013-03-07 17:00:46.939423] D [client.c:2115:client_rpc_notify] 0-gv13-client-0: disconnected (skipped notify) >[2013-03-07 17:00:46.939427] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-0: attempting reconnect >[2013-03-07 17:00:46.939432] D [name.c:155:client_fill_address_family] 0-gv13-client-0: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.939437] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-0: option remote-port missing in volume gv13-client-0. Defaulting to 24007 >[2013-03-07 17:00:46.939442] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:46.939446] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.941713] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.941730] T [socket.c:731:__socket_nodelay] 0-gv13-client-0: NODELAY enabled for socket 7 >[2013-03-07 17:00:46.941737] T [socket.c:790:__socket_keepalive] 0-gv13-client-0: Keep-alive enabled for socket 7, interval 2, idle: 20 >[2013-03-07 17:00:46.941749] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.941793] D [socket.c:486:__socket_rwv] 0-gv13-client-1: EOF on socket >[2013-03-07 17:00:46.941805] W [socket.c:514:__socket_rwv] 0-gv13-client-1: readv failed (No data available) >[2013-03-07 17:00:46.941813] D [socket.c:1962:__socket_proto_state_machine] 0-gv13-client-1: reading from socket failed. Error (No data available), peer (172.22.6.137:24007) >[2013-03-07 17:00:46.941826] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:46.941837] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv13-client-1: cleaning up state in transport object 0x9ad8d0 >[2013-03-07 17:00:46.941844] D [client.c:2115:client_rpc_notify] 0-gv13-client-1: disconnected (skipped notify) >[2013-03-07 17:00:46.941848] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-1: attempting reconnect >[2013-03-07 17:00:46.941853] D [name.c:155:client_fill_address_family] 0-gv13-client-1: address-family not specified, guessing it to be inet from (remote-host: host14) >[2013-03-07 17:00:46.941858] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-1: option remote-port missing in volume gv13-client-1. Defaulting to 24007 >[2013-03-07 17:00:46.941863] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:46.941867] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14 >[2013-03-07 17:00:46.944056] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007 >[2013-03-07 17:00:46.944069] T [socket.c:731:__socket_nodelay] 0-gv13-client-1: NODELAY enabled for socket 9 >[2013-03-07 17:00:46.944075] T [socket.c:790:__socket_keepalive] 0-gv13-client-1: Keep-alive enabled for socket 9, interval 2, idle: 20 >[2013-03-07 17:00:46.944086] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.944359] D [socket.c:486:__socket_rwv] 0-gv3-client-0: EOF on socket >[2013-03-07 17:00:46.944407] W [socket.c:514:__socket_rwv] 0-gv3-client-0: readv failed (No data available) >[2013-03-07 17:00:46.944425] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-0: reading from socket failed. Error (No data available), peer (172.22.6.136:24007) >[2013-03-07 17:00:46.944442] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:46.944481] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-0: cleaning up state in transport object 0xa61220 >[2013-03-07 17:00:46.944529] D [client.c:2115:client_rpc_notify] 0-gv3-client-0: disconnected (skipped notify) >[2013-03-07 17:00:46.944546] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect >[2013-03-07 17:00:46.944563] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.944578] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007 >[2013-03-07 17:00:46.944592] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:46.944605] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.947103] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.947123] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 10 >[2013-03-07 17:00:46.947130] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 10, interval 2, idle: 20 >[2013-03-07 17:00:46.947142] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.947193] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.947202] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv14-client-1: changing port to 49153 (from 0) >[2013-03-07 17:00:46.947220] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.947227] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv3-client-1: changing port to 49155 (from 0) >[2013-03-07 17:00:46.947246] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.947253] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv3-client-2: changing port to 49156 (from 0) >[2013-03-07 17:00:46.947266] D [socket.c:486:__socket_rwv] 0-gv14-client-1: EOF on socket >[2013-03-07 17:00:46.947273] W [socket.c:514:__socket_rwv] 0-gv14-client-1: readv failed (No data available) >[2013-03-07 17:00:46.947278] D [socket.c:1962:__socket_proto_state_machine] 0-gv14-client-1: reading from socket failed. Error (No data available), peer (172.22.6.136:24007) >[2013-03-07 17:00:46.947283] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:46.947292] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv14-client-1: cleaning up state in transport object 0xaa7330 >[2013-03-07 17:00:46.947299] D [client.c:2115:client_rpc_notify] 0-gv14-client-1: disconnected (skipped notify) >[2013-03-07 17:00:46.947304] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-1: attempting reconnect >[2013-03-07 17:00:46.947310] D [name.c:155:client_fill_address_family] 0-gv14-client-1: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.947314] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-1: option remote-port missing in volume gv14-client-1. Defaulting to 24007 >[2013-03-07 17:00:46.947319] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:46.947323] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.949575] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.949593] T [socket.c:731:__socket_nodelay] 0-gv14-client-1: NODELAY enabled for socket 11 >[2013-03-07 17:00:46.949599] T [socket.c:790:__socket_keepalive] 0-gv14-client-1: Keep-alive enabled for socket 11, interval 2, idle: 20 >[2013-03-07 17:00:46.949611] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.949655] D [socket.c:486:__socket_rwv] 0-gv3-client-1: EOF on socket >[2013-03-07 17:00:46.949664] W [socket.c:514:__socket_rwv] 0-gv3-client-1: readv failed (No data available) >[2013-03-07 17:00:46.949669] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-1: reading from socket failed. Error (No data available), peer (172.22.6.136:24007) >[2013-03-07 17:00:46.949673] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:46.949682] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-1: cleaning up state in transport object 0xa3ca30 >[2013-03-07 17:00:46.949689] D [client.c:2115:client_rpc_notify] 0-gv3-client-1: disconnected (skipped notify) >[2013-03-07 17:00:46.949694] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-1: attempting reconnect >[2013-03-07 17:00:46.949699] D [name.c:155:client_fill_address_family] 0-gv3-client-1: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.949704] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-1: option remote-port missing in volume gv3-client-1. Defaulting to 24007 >[2013-03-07 17:00:46.949708] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:46.949712] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.951924] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.951937] T [socket.c:731:__socket_nodelay] 0-gv3-client-1: NODELAY enabled for socket 12 >[2013-03-07 17:00:46.951943] T [socket.c:790:__socket_keepalive] 0-gv3-client-1: Keep-alive enabled for socket 12, interval 2, idle: 20 >[2013-03-07 17:00:46.951953] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.951997] D [socket.c:486:__socket_rwv] 0-gv3-client-2: EOF on socket >[2013-03-07 17:00:46.952005] W [socket.c:514:__socket_rwv] 0-gv3-client-2: readv failed (No data available) >[2013-03-07 17:00:46.952009] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-2: reading from socket failed. Error (No data available), peer (172.22.6.136:24007) >[2013-03-07 17:00:46.952014] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:46.952023] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-2: cleaning up state in transport object 0xa181e0 >[2013-03-07 17:00:46.952029] D [client.c:2115:client_rpc_notify] 0-gv3-client-2: disconnected (skipped notify) >[2013-03-07 17:00:46.952034] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-2: attempting reconnect >[2013-03-07 17:00:46.952039] D [name.c:155:client_fill_address_family] 0-gv3-client-2: address-family not specified, guessing it to be inet from (remote-host: host13) >[2013-03-07 17:00:46.952043] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-2: option remote-port missing in volume gv3-client-2. Defaulting to 24007 >[2013-03-07 17:00:46.952048] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:46.952052] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13 >[2013-03-07 17:00:46.954227] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007 >[2013-03-07 17:00:46.954241] T [socket.c:731:__socket_nodelay] 0-gv3-client-2: NODELAY enabled for socket 13 >[2013-03-07 17:00:46.954247] T [socket.c:790:__socket_keepalive] 0-gv3-client-2: Keep-alive enabled for socket 13, interval 2, idle: 20 >[2013-03-07 17:00:46.954258] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.954301] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.954310] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv14-client-0: changing port to 49153 (from 0) >[2013-03-07 17:00:46.954323] D [client.c:2050:client_rpc_notify] 0-gv13-client-0: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.954331] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.954337] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.954358] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.954379] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.954391] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.954408] D [client.c:2050:client_rpc_notify] 0-gv13-client-1: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.954418] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.954427] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.954454] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.954470] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.954484] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.954494] D [client.c:2050:client_rpc_notify] 0-gv3-client-0: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.954501] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.954507] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.954524] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.954543] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.954550] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.954567] D [socket.c:486:__socket_rwv] 0-gv14-client-0: EOF on socket >[2013-03-07 17:00:46.954573] W [socket.c:514:__socket_rwv] 0-gv14-client-0: readv failed (No data available) >[2013-03-07 17:00:46.954578] D [socket.c:1962:__socket_proto_state_machine] 0-gv14-client-0: reading from socket failed. Error (No data available), peer (172.22.6.137:24007) >[2013-03-07 17:00:46.954582] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now >[2013-03-07 17:00:46.954593] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv14-client-0: cleaning up state in transport object 0xacbb80 >[2013-03-07 17:00:46.954599] D [client.c:2115:client_rpc_notify] 0-gv14-client-0: disconnected (skipped notify) >[2013-03-07 17:00:46.954603] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-0: attempting reconnect >[2013-03-07 17:00:46.954608] D [name.c:155:client_fill_address_family] 0-gv14-client-0: address-family not specified, guessing it to be inet from (remote-host: host14) >[2013-03-07 17:00:46.954613] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-0: option remote-port missing in volume gv14-client-0. Defaulting to 24007 >[2013-03-07 17:00:46.954618] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache >[2013-03-07 17:00:46.954622] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14 >[2013-03-07 17:00:46.956851] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007 >[2013-03-07 17:00:46.956866] T [socket.c:731:__socket_nodelay] 0-gv14-client-0: NODELAY enabled for socket 14 >[2013-03-07 17:00:46.956872] T [socket.c:790:__socket_keepalive] 0-gv14-client-0: Keep-alive enabled for socket 14, interval 2, idle: 20 >[2013-03-07 17:00:46.956883] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket: is not a valid port identifier >[2013-03-07 17:00:46.957166] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.957217] T [client-handshake.c:1664:select_server_supported_programs] 0-gv13-client-0: GF-DUMP (1) not supported >[2013-03-07 17:00:46.957234] I [client-handshake.c:1658:select_server_supported_programs] 0-gv13-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-03-07 17:00:46.957293] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.957320] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64 >[2013-03-07 17:00:46.957403] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.957452] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.957476] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.957510] D [client.c:2050:client_rpc_notify] 0-gv14-client-1: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.957531] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.957548] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.957601] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.957684] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.957708] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.957738] D [client.c:2050:client_rpc_notify] 0-gv3-client-1: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.957757] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.957774] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.957821] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.957870] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.957891] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.957920] D [client.c:2050:client_rpc_notify] 0-gv3-client-2: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.957939] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.957955] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.957998] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected >[2013-03-07 17:00:46.958050] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.958085] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.958153] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.958186] T [client-handshake.c:1664:select_server_supported_programs] 0-gv14-client-1: GF-DUMP (1) not supported >[2013-03-07 17:00:46.958209] I [client-handshake.c:1658:select_server_supported_programs] 0-gv14-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-03-07 17:00:46.958267] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.958293] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64 >[2013-03-07 17:00:46.958365] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.958425] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.958455] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.958508] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.958531] T [client-handshake.c:1664:select_server_supported_programs] 0-gv13-client-1: GF-DUMP (1) not supported >[2013-03-07 17:00:46.958544] I [client-handshake.c:1658:select_server_supported_programs] 0-gv13-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-03-07 17:00:46.958581] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.958599] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64 >[2013-03-07 17:00:46.958658] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.958690] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.958710] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.958742] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.958763] T [client-handshake.c:1664:select_server_supported_programs] 0-gv3-client-0: GF-DUMP (1) not supported >[2013-03-07 17:00:46.958776] I [client-handshake.c:1658:select_server_supported_programs] 0-gv3-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-03-07 17:00:46.958810] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.958827] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64 >[2013-03-07 17:00:46.958871] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.958912] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.958960] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.959005] D [client.c:2050:client_rpc_notify] 0-gv14-client-0: got RPC_CLNT_CONNECT >[2013-03-07 17:00:46.959034] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.959060] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 >[2013-03-07 17:00:46.959123] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.959173] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.959201] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.959256] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.959345] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv13-client-1: clnt-lk-version = 1, server-lk-version = 0 >[2013-03-07 17:00:46.959376] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv13-client-1: Connected to 172.22.6.137:49152, attached to remote volume '/gfs0/brick13'. >[2013-03-07 17:00:46.959425] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv13-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2013-03-07 17:00:46.959454] D [client-handshake.c:1318:client_post_handshake] 0-gv13-client-1: No fds to open - notifying all parents child up >[2013-03-07 17:00:46.959481] D [client-handshake.c:486:client_set_lk_version] 0-gv13-client-1: Sending SET_LK_VERSION >[2013-03-07 17:00:46.959512] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.959542] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 >[2013-03-07 17:00:46.959587] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.959643] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.959671] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64 >[2013-03-07 17:00:46.959713] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.959747] I [afr-common.c:3724:afr_notify] 0-gv13-replicate-0: Subvolume 'gv13-client-1' came back up; going online. >[2013-03-07 17:00:46.959874] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.959921] T [client-handshake.c:1664:select_server_supported_programs] 0-gv3-client-1: GF-DUMP (1) not supported >[2013-03-07 17:00:46.959930] I [client-handshake.c:1658:select_server_supported_programs] 0-gv3-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-03-07 17:00:46.959961] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.959997] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64 >[2013-03-07 17:00:46.959994] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.960062] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.960078] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.960113] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.960140] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.960175] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.960222] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.960263] T [client-handshake.c:1664:select_server_supported_programs] 0-gv3-client-2: GF-DUMP (1) not supported >[2013-03-07 17:00:46.960293] I [client-handshake.c:1658:select_server_supported_programs] 0-gv3-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-03-07 17:00:46.960350] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.960387] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64 >[2013-03-07 17:00:46.960459] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected >[2013-03-07 17:00:46.960517] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.960551] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.960609] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.960656] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv13-client-1: Server lk version = 1 >[2013-03-07 17:00:46.960697] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.960727] T [client-handshake.c:1664:select_server_supported_programs] 0-gv14-client-0: GF-DUMP (1) not supported >[2013-03-07 17:00:46.960749] I [client-handshake.c:1658:select_server_supported_programs] 0-gv14-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-03-07 17:00:46.960803] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.960831] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64 >[2013-03-07 17:00:46.960912] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.960957] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.960987] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1) >[2013-03-07 17:00:46.961038] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.961088] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.961200] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-1 is remote >[2013-03-07 17:00:46.961235] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.961249] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.961289] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client3_3_getxattr+0x3bb) [0x7fe879bd349b] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected >[2013-03-07 17:00:46.961316] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.961329] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (2 || 1) >[2013-03-07 17:00:46.962779] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.962853] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv14-client-0: clnt-lk-version = 1, server-lk-version = 0 >[2013-03-07 17:00:46.962882] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv14-client-0: Connected to 172.22.6.137:49153, attached to remote volume '/gfs0/brick14'. >[2013-03-07 17:00:46.962906] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv14-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2013-03-07 17:00:46.962929] D [client-handshake.c:1318:client_post_handshake] 0-gv14-client-0: No fds to open - notifying all parents child up >[2013-03-07 17:00:46.962959] D [client-handshake.c:486:client_set_lk_version] 0-gv14-client-0: Sending SET_LK_VERSION >[2013-03-07 17:00:46.962990] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.963013] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 >[2013-03-07 17:00:46.963049] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.963074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.963091] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64 >[2013-03-07 17:00:46.963146] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.963170] I [afr-common.c:3724:afr_notify] 0-gv14-replicate-0: Subvolume 'gv14-client-0' came back up; going online. >[2013-03-07 17:00:46.963251] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.963289] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv13-client-0: clnt-lk-version = 1, server-lk-version = 0 >[2013-03-07 17:00:46.963304] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv13-client-0: Connected to 172.22.6.136:49152, attached to remote volume '/gfs0/brick13'. >[2013-03-07 17:00:46.963318] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv13-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2013-03-07 17:00:46.963330] D [client-handshake.c:1318:client_post_handshake] 0-gv13-client-0: No fds to open - notifying all parents child up >[2013-03-07 17:00:46.963326] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.963348] D [client-handshake.c:486:client_set_lk_version] 0-gv13-client-0: Sending SET_LK_VERSION >[2013-03-07 17:00:46.963387] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.963403] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.963446] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 >[2013-03-07 17:00:46.963448] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.963494] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.963556] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.963574] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64 >[2013-03-07 17:00:46.963609] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.963688] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.963696] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv14-client-0: Server lk version = 1 >[2013-03-07 17:00:46.963696] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.963705] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.963712] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.963723] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.963734] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv14-client-1: clnt-lk-version = 1, server-lk-version = 0 >[2013-03-07 17:00:46.963733] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.963738] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv14-client-1: Connected to 172.22.6.136:49153, attached to remote volume '/gfs0/brick14'. >[2013-03-07 17:00:46.963737] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-0 is local >[2013-03-07 17:00:46.963755] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv14-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2013-03-07 17:00:46.963808] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv13-replicate-0: Setting lk-owner=140636503968112 >[2013-03-07 17:00:46.963831] D [client-handshake.c:1318:client_post_handshake] 0-gv14-client-1: No fds to open - notifying all parents child up >[2013-03-07 17:00:46.963855] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv13-replicate-0: starting crawl 1 for gv13-client-0 >[2013-03-07 17:00:46.963886] D [client-handshake.c:486:client_set_lk_version] 0-gv14-client-1: Sending SET_LK_VERSION >[2013-03-07 17:00:46.963911] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.963929] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 >[2013-03-07 17:00:46.963974] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.963989] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964000] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964021] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.964032] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64 >[2013-03-07 17:00:46.964064] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.964064] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.964071] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.964097] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.964124] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964131] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv13-client-0: Server lk version = 1 >[2013-03-07 17:00:46.964082] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.964135] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964160] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964143] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.964169] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.964180] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv14-client-1: Server lk version = 1 >[2013-03-07 17:00:46.964207] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.964240] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.964251] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-0 is remote >[2013-03-07 17:00:46.964257] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964339] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964345] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.964349] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.964358] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.964364] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.964369] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv3-client-0: clnt-lk-version = 1, server-lk-version = 0 >[2013-03-07 17:00:46.964374] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv3-client-0: Connected to 172.22.6.136:49154, attached to remote volume '/gfs0/brick1'. >[2013-03-07 17:00:46.964379] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv3-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2013-03-07 17:00:46.964384] D [client-handshake.c:1318:client_post_handshake] 0-gv3-client-0: No fds to open - notifying all parents child up >[2013-03-07 17:00:46.964390] D [client-handshake.c:486:client_set_lk_version] 0-gv3-client-0: Sending SET_LK_VERSION >[2013-03-07 17:00:46.964445] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964473] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 >[2013-03-07 17:00:46.964489] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.964498] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964504] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64 >[2013-03-07 17:00:46.964514] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.964521] I [afr-common.c:3724:afr_notify] 0-gv3-replicate-0: Subvolume 'gv3-client-0' came back up; going online. >[2013-03-07 17:00:46.964548] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964560] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964563] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-0 is local >[2013-03-07 17:00:46.964566] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.964584] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv13-replicate-0: Setting lk-owner=140636503957692 >[2013-03-07 17:00:46.964576] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.964597] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964595] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv13-replicate-0: starting crawl 1 for gv13-client-0 >[2013-03-07 17:00:46.964615] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.964631] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.964633] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-1 is remote >[2013-03-07 17:00:46.964645] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv3-client-0: Server lk version = 1 >[2013-03-07 17:00:46.964644] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964665] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.964666] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.964683] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.964689] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.964694] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964703] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.964709] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv13-replicate-0: Another crawl is in progress for gv13-client-0 >[2013-03-07 17:00:46.964723] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964744] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-1 is local >[2013-03-07 17:00:46.964757] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv14-replicate-0: Setting lk-owner=140636503959776 >[2013-03-07 17:00:46.964763] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv14-replicate-0: starting crawl 1 for gv14-client-1 >[2013-03-07 17:00:46.964760] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.964789] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-0 is remote >[2013-03-07 17:00:46.964790] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964811] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964823] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.964857] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.964862] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.964871] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-0 is local >[2013-03-07 17:00:46.964874] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.964880] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv13-replicate-0: Setting lk-owner=140636503972280 >[2013-03-07 17:00:46.964891] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.964894] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv13-replicate-0: starting crawl 1 for gv13-client-0 >[2013-03-07 17:00:46.964938] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.964945] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.964952] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.964981] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.964983] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965002] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local >[2013-03-07 17:00:46.965013] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503978532 >[2013-03-07 17:00:46.965012] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.965023] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0 >[2013-03-07 17:00:46.965032] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.965050] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.965073] D [afr-self-heald.c:1000:_crawl_directory] 0-gv13-replicate-0: crawling INDEX 51a9e1e3-32e0-4bd4-94ae-7ea06c6cf9ae >[2013-03-07 17:00:46.965074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.965102] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.965107] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.965116] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.965122] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.965136] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.965138] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv13-replicate-0: Another crawl is in progress for gv13-client-0 >[2013-03-07 17:00:46.965173] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.965184] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.965183] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965197] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.965209] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-1 is local >[2013-03-07 17:00:46.965222] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965224] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv14-replicate-0: Setting lk-owner=140636503970196 >[2013-03-07 17:00:46.965239] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv14-replicate-0: starting crawl 1 for gv14-client-1 >[2013-03-07 17:00:46.965251] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.965263] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.965293] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965365] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.965379] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv14-replicate-0: Another crawl is in progress for gv14-client-1 >[2013-03-07 17:00:46.965393] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-0) >[2013-03-07 17:00:46.965406] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-0 is remote >[2013-03-07 17:00:46.965407] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local >[2013-03-07 17:00:46.965437] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.965438] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.965466] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.965466] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.965479] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.965448] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503972280 >[2013-03-07 17:00:46.965482] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965503] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0 >[2013-03-07 17:00:46.965509] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.965517] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.965538] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.965555] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.965563] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.965567] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-1) >[2013-03-07 17:00:46.965570] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.965602] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.965609] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965573] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.965622] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.965663] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client3_3_getxattr+0x3bb) [0x7fe879bd349b] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected >[2013-03-07 17:00:46.965665] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.965685] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.965687] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-0 >[2013-03-07 17:00:46.965698] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (2 || 1) >[2013-03-07 17:00:46.965717] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-1 is remote >[2013-03-07 17:00:46.965726] D [afr-self-heald.c:1000:_crawl_directory] 0-gv14-replicate-0: crawling INDEX c34b72c4-a2ff-4056-bea4-6d8eb09a45d9 >[2013-03-07 17:00:46.965718] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965743] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.965753] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.965757] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-1 is local >[2013-03-07 17:00:46.965763] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.965771] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv14-replicate-0: Setting lk-owner=140636503974364 >[2013-03-07 17:00:46.965773] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.965784] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv14-replicate-0: starting crawl 1 for gv14-client-1 >[2013-03-07 17:00:46.965781] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv3-client-1: clnt-lk-version = 1, server-lk-version = 0 >[2013-03-07 17:00:46.965776] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv13-replicate-0: Crawl completed on gv13-client-0 >[2013-03-07 17:00:46.965797] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv3-client-1: Connected to 172.22.6.136:49155, attached to remote volume '/gfs0/brick2'. >[2013-03-07 17:00:46.965841] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv3-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2013-03-07 17:00:46.965852] D [client-handshake.c:1318:client_post_handshake] 0-gv3-client-1: No fds to open - notifying all parents child up >[2013-03-07 17:00:46.965873] D [client-handshake.c:486:client_set_lk_version] 0-gv3-client-1: Sending SET_LK_VERSION >[2013-03-07 17:00:46.965877] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.965888] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.965909] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.965918] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 >[2013-03-07 17:00:46.965980] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.965987] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.966005] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.966035] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64 >[2013-03-07 17:00:46.966048] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.966195] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.966228] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.966244] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.966259] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.966272] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.966315] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.966314] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.966399] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.966403] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.966446] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.966453] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.966462] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local >[2013-03-07 17:00:46.966465] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv3-client-2: clnt-lk-version = 1, server-lk-version = 0 >[2013-03-07 17:00:46.966469] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503963944 >[2013-03-07 17:00:46.966471] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv3-client-2: Connected to 172.22.6.136:49156, attached to remote volume '/gfs0/brick3'. >[2013-03-07 17:00:46.966477] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1 >[2013-03-07 17:00:46.966473] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.966482] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv3-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2013-03-07 17:00:46.966513] D [client-handshake.c:1318:client_post_handshake] 0-gv3-client-2: No fds to open - notifying all parents child up >[2013-03-07 17:00:46.966524] D [client-handshake.c:486:client_set_lk_version] 0-gv3-client-2: Sending SET_LK_VERSION >[2013-03-07 17:00:46.966533] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.966539] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64 >[2013-03-07 17:00:46.966556] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.966530] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac >[2013-03-07 17:00:46.966570] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.966598] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64 >[2013-03-07 17:00:46.966601] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.966616] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.966623] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.966677] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.966744] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.966778] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.966773] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.966800] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.966816] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv3-client-1: Server lk version = 1 >[2013-03-07 17:00:46.966814] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.966826] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.966867] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv3-client-2: Server lk version = 1 >[2013-03-07 17:00:46.966882] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.966916] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.966939] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.966940] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.966956] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.966961] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.966983] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.967005] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.967020] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.967026] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.967052] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv14-replicate-0: Crawl completed on gv14-client-1 >[2013-03-07 17:00:46.967097] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.967115] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.967154] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.967154] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.967215] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.967231] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.967236] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.967285] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.967317] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local >[2013-03-07 17:00:46.967345] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503986868 >[2013-03-07 17:00:46.967360] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1 >[2013-03-07 17:00:46.967373] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.967491] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.967833] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.967865] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.967911] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.967939] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0 >[2013-03-07 17:00:46.967976] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.967994] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.968032] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.968070] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv14-replicate-0: Another crawl is in progress for gv14-client-1 >[2013-03-07 17:00:46.968130] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local >[2013-03-07 17:00:46.968148] D [afr-self-heald.c:1000:_crawl_directory] 0-gv13-replicate-0: crawling INDEX 51a9e1e3-32e0-4bd4-94ae-7ea06c6cf9ae >[2013-03-07 17:00:46.968192] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.968196] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503970196 >[2013-03-07 17:00:46.968190] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.968226] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.968349] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.968254] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.968293] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2 >[2013-03-07 17:00:46.968386] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.968247] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.968351] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.968479] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.968526] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.968527] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.968560] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.968573] D [afr-self-heald.c:1000:_crawl_directory] 0-gv14-replicate-0: crawling INDEX c34b72c4-a2ff-4056-bea4-6d8eb09a45d9 >[2013-03-07 17:00:46.968585] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.968627] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.968633] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.968664] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.968678] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.968657] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.968678] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.968759] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.968683] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-1 >[2013-03-07 17:00:46.968656] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.968726] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.968728] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.968821] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.969122] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.969153] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local >[2013-03-07 17:00:46.969130] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.969184] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.968826] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.969180] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503963944 >[2013-03-07 17:00:46.969213] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.969217] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0 >[2013-03-07 17:00:46.969247] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.969230] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.969264] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.969282] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.969279] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.969288] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.969317] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac >[2013-03-07 17:00:46.969320] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.969332] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.969340] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.969343] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.969349] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.969349] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.969396] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.969395] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.969394] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.969433] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.969462] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.969476] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-0 >[2013-03-07 17:00:46.969501] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.969472] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.969549] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0) >[2013-03-07 17:00:46.969522] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.969599] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.969603] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv13-replicate-0: Crawl completed on gv13-client-0 >[2013-03-07 17:00:46.969532] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.969494] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.969609] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.969712] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.969623] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.969540] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local >[2013-03-07 17:00:46.969748] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.969765] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.969752] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.969811] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.969772] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503976448 >[2013-03-07 17:00:46.969816] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.969828] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0 >[2013-03-07 17:00:46.969837] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.969853] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.969860] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.969864] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv14-replicate-0: Crawl completed on gv14-client-1 >[2013-03-07 17:00:46.969921] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.969894] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local >[2013-03-07 17:00:46.969976] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.970002] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX cfb7526b-cb8e-4c9a-90c1-0009a48b9d27 >[2013-03-07 17:00:46.970031] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.969983] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503982700 >[2013-03-07 17:00:46.970022] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.970065] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.970071] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2 >[2013-03-07 17:00:46.970124] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.970085] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0 >[2013-03-07 17:00:46.970151] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.970080] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.970190] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.970221] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.970242] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.970252] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX 484a792f-dd0b-4c99-9d77-3cb20c96c621 >[2013-03-07 17:00:46.970294] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local >[2013-03-07 17:00:46.970260] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.970299] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.970205] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-2 >[2013-03-07 17:00:46.970393] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.970301] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.970325] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503953524 >[2013-03-07 17:00:46.970374] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.970485] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.970494] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.970531] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.970525] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.970483] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1 >[2013-03-07 17:00:46.970535] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.970617] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.970645] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=78a027b5-dae3-4851-904c-926034438fb4, count=2 >[2013-03-07 17:00:46.970645] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.970695] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.970702] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-1 >[2013-03-07 17:00:46.970702] D [afr-self-heald.c:412:_self_heal_entry] 0-gv3-replicate-0: lookup <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.970722] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.970722] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.970717] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.970814] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.970830] D [afr-self-heald.c:1000:_crawl_directory] 0-gv14-replicate-0: crawling INDEX c34b72c4-a2ff-4056-bea4-6d8eb09a45d9 >[2013-03-07 17:00:46.970835] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.970872] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.970835] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-gv3-replicate-0: <gfid:78a027b5-dae3-4851-904c-926034438fb4>: failed to get the gfid from dict >[2013-03-07 17:00:46.970892] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.970929] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.970962] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.970971] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.970981] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.971026] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.971023] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68 >[2013-03-07 17:00:46.971036] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.971037] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.971099] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=78a027b5-dae3-4851-904c-926034438fb4, count=2 >[2013-03-07 17:00:46.971112] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971150] D [afr-self-heald.c:412:_self_heal_entry] 0-gv3-replicate-0: lookup <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.971157] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971180] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.971199] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-gv3-replicate-0: <gfid:78a027b5-dae3-4851-904c-926034438fb4>: failed to get the gfid from dict >[2013-03-07 17:00:46.971200] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.971206] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68 >[2013-03-07 17:00:46.971217] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.971218] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.971216] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac >[2013-03-07 17:00:46.971262] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.971251] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68 >[2013-03-07 17:00:46.971299] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.971241] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.971321] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.971325] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971356] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.971325] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.971378] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.971359] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68 >[2013-03-07 17:00:46.971388] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68 >[2013-03-07 17:00:46.971408] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971384] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000 >[2013-03-07 17:00:46.971429] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.971443] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.971461] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971466] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local >[2013-03-07 17:00:46.971488] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.971503] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.971576] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971553] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503963944 >[2013-03-07 17:00:46.971584] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68 >[2013-03-07 17:00:46.971598] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-0 >[2013-03-07 17:00:46.971640] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.971611] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2 >[2013-03-07 17:00:46.971585] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.971613] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.971629] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971781] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.971787] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.971810] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1) >[2013-03-07 17:00:46.971814] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.971824] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.971840] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.971833] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv14-replicate-0: Crawl completed on gv14-client-1 >[2013-03-07 17:00:46.971954] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.972040] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.972089] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.972093] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.972086] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0 >[2013-03-07 17:00:46.972165] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.972217] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.972218] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.972246] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.972277] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.972314] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.972353] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.972368] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.972373] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 1 >[2013-03-07 17:00:46.972443] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 1 for <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.972455] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 1 >[2013-03-07 17:00:46.972463] D [afr-common.c:1282:afr_detect_self_heal_by_iatt] 0-gv3-replicate-0: size differs for <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.972469] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972474] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972479] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972486] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972493] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.972499] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.972504] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.972509] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.972515] D [afr-common.c:1443:afr_launch_self_heal] 0-gv3-replicate-0: background meta-data data self-heal triggered. path: <gfid:78a027b5-dae3-4851-904c-926034438fb4>, reason: lookup detected pending operations >[2013-03-07 17:00:46.972524] T [afr-self-heal-common.c:2267:afr_self_heal] 0-gv3-replicate-0: performing self heal on <gfid:78a027b5-dae3-4851-904c-926034438fb4> (metadata=1 data=1 entry=0) >[2013-03-07 17:00:46.972545] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503968112 >[2013-03-07 17:00:46.972580] T [afr-self-heal-common.c:2357:afr_self_heal] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.972609] T [afr-self-heal-common.c:1027:afr_sh_missing_entries_done] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.972661] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.972681] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.972734] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.972769] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.972786] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.972823] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.972835] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.972841] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.972854] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.972873] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.972893] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.972907] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.972913] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.972920] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.972925] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.972930] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 2 >[2013-03-07 17:00:46.972935] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 2 for <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.972941] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 2 >[2013-03-07 17:00:46.972947] D [afr-common.c:1282:afr_detect_self_heal_by_iatt] 0-gv3-replicate-0: size differs for <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.972953] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972958] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972963] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972969] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.972975] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.972993] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.973002] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.973007] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.973012] D [afr-common.c:1443:afr_launch_self_heal] 0-gv3-replicate-0: background meta-data data self-heal triggered. path: <gfid:78a027b5-dae3-4851-904c-926034438fb4>, reason: lookup detected pending operations >[2013-03-07 17:00:46.973018] T [afr-self-heal-common.c:2267:afr_self_heal] 0-gv3-replicate-0: performing self heal on <gfid:78a027b5-dae3-4851-904c-926034438fb4> (metadata=1 data=1 entry=0) >[2013-03-07 17:00:46.973024] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503957692 >[2013-03-07 17:00:46.973038] T [afr-self-heal-common.c:2357:afr_self_heal] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.973070] T [afr-self-heal-common.c:1027:afr_sh_missing_entries_done] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.973097] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.973114] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.973157] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.973189] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.973206] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.973242] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.973253] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.973259] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.973273] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.973288] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.973310] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.973318] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.973324] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.973331] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.973348] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.973360] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.973356] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local >[2013-03-07 17:00:46.973397] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503959776 >[2013-03-07 17:00:46.973429] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1 >[2013-03-07 17:00:46.973377] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.973525] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received >[2013-03-07 17:00:46.973524] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.973524] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-1 >[2013-03-07 17:00:46.973581] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64 >[2013-03-07 17:00:46.973559] T [afr-lk-common.c:1477:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: All servers locked. Calling the cbk >[2013-03-07 17:00:46.973626] D [afr-self-heal-metadata.c:486:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking metadata inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. Proceeding to FOP >[2013-03-07 17:00:46.973662] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.973668] D [afr-self-heal-common.c:1930:afr_sh_common_lookup] 0-gv3-replicate-0: looking up <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-0 >[2013-03-07 17:00:46.973753] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.973761] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 252, payload: 184, rpc hdr: 68 >[2013-03-07 17:00:46.973784] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.973794] D [afr-self-heal-common.c:1930:afr_sh_common_lookup] 0-gv3-replicate-0: looking up <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-1 >[2013-03-07 17:00:46.973804] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.973810] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 252, payload: 184, rpc hdr: 68 >[2013-03-07 17:00:46.973828] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.973836] D [afr-self-heal-common.c:1930:afr_sh_common_lookup] 0-gv3-replicate-0: looking up <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-2 >[2013-03-07 17:00:46.973847] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.973853] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 252, payload: 184, rpc hdr: 68 >[2013-03-07 17:00:46.973867] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.973886] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.973898] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.973910] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.973924] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received >[2013-03-07 17:00:46.973930] T [afr-lk-common.c:1486:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: 0 servers locked. Trying again with blocking calls >[2013-03-07 17:00:46.973937] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.973944] T [afr-lk-common.c:716:afr_unlock_inodelk] 0-gv3-replicate-0: No internal locks unlocked >[2013-03-07 17:00:46.973950] D [afr-self-heal-metadata.c:478:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking metadata inodelks failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.973956] D [afr-self-heal-metadata.c:480:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Metadata self-heal failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. >[2013-03-07 17:00:46.973961] D [afr-self-heal-metadata.c:61:afr_sh_metadata_done] 0-gv3-replicate-0: proceeding to data check on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.973979] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.973986] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68 >[2013-03-07 17:00:46.974001] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.974044] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.974060] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68 >[2013-03-07 17:00:46.974094] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.974106] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.974112] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68 >[2013-03-07 17:00:46.974126] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.974141] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.974160] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.974164] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac >[2013-03-07 17:00:46.974179] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.974178] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local >[2013-03-07 17:00:46.974188] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.974183] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.974194] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503953524 >[2013-03-07 17:00:46.974205] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2 >[2013-03-07 17:00:46.974205] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.974212] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.974261] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.974286] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-0 >[2013-03-07 17:00:46.974434] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.974475] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.974496] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.974510] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.974525] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.974540] T [afr-self-heal-metadata.c:355:afr_sh_metadata_sync_prepare] 0-gv3-replicate-0: syncing metadata of <gfid:78a027b5-dae3-4851-904c-926034438fb4> from subvolume gv3-client-1 to 1 active sinks >[2013-03-07 17:00:46.974574] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.974596] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.974640] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-2 >[2013-03-07 17:00:46.974659] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.974712] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.974738] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-1 >[2013-03-07 17:00:46.974789] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.974817] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.974832] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.974886] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.974911] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-2 >[2013-03-07 17:00:46.974907] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000 >[2013-03-07 17:00:46.974927] T [afr-self-heal-data.c:1280:afr_sh_data_open_cbk] 0-gv3-replicate-0: fd for <gfid:78a027b5-dae3-4851-904c-926034438fb4> opened, commencing sync >[2013-03-07 17:00:46.974955] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.974987] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.975009] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.975015] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.975105] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.975160] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.975178] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.975220] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.975250] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.975265] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.975301] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.975352] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.975364] D [afr-self-heal-metadata.c:267:afr_sh_metadata_sync] 0-gv3-replicate-0: self-healing metadata of <gfid:78a027b5-dae3-4851-904c-926034438fb4> from gv3-client-1 to gv3-client-0 >[2013-03-07 17:00:46.975376] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.975382] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68 >[2013-03-07 17:00:46.975398] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.975418] T [client.c:1176:is_client_rpc_init_command] 0-gv3-client-0: name is !replace-brick >[2013-03-07 17:00:46.975425] T [client.c:1201:is_client_rpc_destroy_command] 0-gv3-client-0: name is !replace-brick >[2013-03-07 17:00:46.975434] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.975440] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 100, payload: 32, rpc hdr: 68 >[2013-03-07 17:00:46.975453] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 17) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.975496] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.975542] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.975541] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0 >[2013-03-07 17:00:46.975579] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.975726] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.975772] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received >[2013-03-07 17:00:46.975789] T [afr-lk-common.c:1486:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: 0 servers locked. Trying again with blocking calls >[2013-03-07 17:00:46.975817] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.975832] T [afr-lk-common.c:716:afr_unlock_inodelk] 0-gv3-replicate-0: No internal locks unlocked >[2013-03-07 17:00:46.975849] D [afr-self-heal-data.c:1134:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. by bc60dc7ce87f0000 >[2013-03-07 17:00:46.975885] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.975904] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.975961] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.976006] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.976049] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 17) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.976090] D [afr-self-heal-common.c:2510:afr_sh_erase_pending] 0-gv3-replicate-0: Delta matrix for: 7089dc7ce87f0000 >[2013-03-07 17:00:46.976107] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.976121] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ] >[2013-03-07 17:00:46.976134] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ] >[2013-03-07 17:00:46.976164] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.976183] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 236, payload: 168, rpc hdr: 68 >[2013-03-07 17:00:46.976223] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.976258] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.976275] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 236, payload: 168, rpc hdr: 68 >[2013-03-07 17:00:46.976315] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.976347] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.976363] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 236, payload: 168, rpc hdr: 68 >[2013-03-07 17:00:46.976400] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.976453] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-2 >[2013-03-07 17:00:46.976591] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.976903] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.976975] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.977015] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.977046] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.977065] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.977123] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.977158] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.977175] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.977217] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.977247] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.977264] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.977303] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.977405] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.977447] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.977472] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.977524] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.977580] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.977618] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.977648] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.977665] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked >[2013-03-07 17:00:46.977685] D [afr-self-heal-metadata.c:61:afr_sh_metadata_done] 0-gv3-replicate-0: proceeding to data check on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.977717] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.977738] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68 >[2013-03-07 17:00:46.977778] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.977809] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.977826] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68 >[2013-03-07 17:00:46.977864] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.977894] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.977923] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68 >[2013-03-07 17:00:46.977960] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.978010] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.978021] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.978027] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.978039] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.978051] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.978059] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-0 >[2013-03-07 17:00:46.978147] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.978189] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-1 >[2013-03-07 17:00:46.978219] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.978239] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-2 >[2013-03-07 17:00:46.978253] T [afr-self-heal-data.c:1280:afr_sh_data_open_cbk] 0-gv3-replicate-0: fd for <gfid:78a027b5-dae3-4851-904c-926034438fb4> opened, commencing sync >[2013-03-07 17:00:46.978278] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.978294] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.978335] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.978368] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.978384] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.978433] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.978465] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.978481] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.978519] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.978559] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.978581] D [afr-lk-common.c:1098:afr_lock_blocking] 0-gv3-replicate-0: we're done locking >[2013-03-07 17:00:46.978593] D [afr-self-heal-data.c:1112:afr_sh_data_post_blocking_inodelk_cbk] 0-gv3-replicate-0: Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by bc60dc7ce87f0000. Proceding to self-heal >[2013-03-07 17:00:46.978610] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.978617] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.978643] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.978657] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.978664] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.978679] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.978716] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.978733] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.978767] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.978812] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.978845] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.978874] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.978891] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received >[2013-03-07 17:00:46.978905] T [afr-lk-common.c:1486:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: 0 servers locked. Trying again with blocking calls >[2013-03-07 17:00:46.978919] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.978931] T [afr-lk-common.c:716:afr_unlock_inodelk] 0-gv3-replicate-0: No internal locks unlocked >[2013-03-07 17:00:46.978947] D [afr-self-heal-data.c:1134:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. by 7089dc7ce87f0000 >[2013-03-07 17:00:46.978971] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.978988] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.979028] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.979079] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.979108] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded >[2013-03-07 17:00:46.979139] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.979171] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded >[2013-03-07 17:00:46.979216] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.979242] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded >[2013-03-07 17:00:46.979273] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.979291] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.979327] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.979356] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.979373] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.979405] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.979462] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.979480] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.979514] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.979691] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.979726] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded >[2013-03-07 17:00:46.979756] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.979775] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded >[2013-03-07 17:00:46.979905] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.979934] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded >[2013-03-07 17:00:46.979952] D [afr-self-heal-data.c:692:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: Pending matrix for: bc60dc7ce87f0000 >[2013-03-07 17:00:46.979974] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.979989] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.980002] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.980015] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.980033] D [afr-self-heal-data.c:672:afr_sh_data_fix] 0-gv3-replicate-0: self-healing file <gfid:78a027b5-dae3-4851-904c-926034438fb4> from subvolume gv3-client-1 to 1 other >[2013-03-07 17:00:46.980071] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.980089] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 104, payload: 36, rpc hdr: 68 >[2013-03-07 17:00:46.980137] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 24) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.980443] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 24) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.980476] D [afr-self-heal-data.c:547:afr_sh_data_trim_cbk] 0-gv3-replicate-0: ftruncate of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-0 completed >[2013-03-07 17:00:46.980519] T [afr-self-heal-algorithm.c:350:sh_loop_driver] 0-gv3-replicate-0: spawning a loop for offset 0 >[2013-03-07 17:00:46.980558] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.980577] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.980620] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.980652] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.980667] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.980706] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.980735] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.980750] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.980787] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.980944] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.981158] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.981191] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.981208] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received >[2013-03-07 17:00:46.981222] T [afr-lk-common.c:1477:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: All servers locked. Calling the cbk >[2013-03-07 17:00:46.981238] D [afr-self-heal-data.c:1147:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by bc60dc7ce87f0000. Proceeding to self-heal >[2013-03-07 17:00:46.981257] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.981280] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.981296] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.981337] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.981378] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.981395] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.981445] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.981476] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.981492] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.981528] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.981549] D [afr-self-heal-algorithm.c:168:sh_loop_lock_success] 0-gv3-replicate-0: Acquired lock for range 0 131072 >[2013-03-07 17:00:46.981579] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.981597] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.981628] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 12) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.981664] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.981924] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.981956] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.981973] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked >[2013-03-07 17:00:46.982763] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 12) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.982793] T [afr-self-heal-algorithm.c:513:sh_loop_read_cbk] 0-gv3-replicate-0: read 1979 bytes of data from <gfid:78a027b5-dae3-4851-904c-926034438fb4>, offset 0 >[2013-03-07 17:00:46.982822] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.982838] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.982884] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.983207] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.983237] T [afr-self-heal-algorithm.c:426:sh_loop_write_cbk] 0-gv3-replicate-0: wrote 1979 bytes of data from <gfid:78a027b5-dae3-4851-904c-926034438fb4> to child 0, offset 0 >[2013-03-07 17:00:46.983256] T [afr-self-heal-algorithm.c:383:sh_loop_return] 0-gv3-replicate-0: loop for offset 0 returned >[2013-03-07 17:00:46.983269] D [afr-self-heal-algorithm.c:115:sh_loop_driver_done] 0-gv3-replicate-0: full self-heal completed on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.983303] D [afr-self-heal-common.c:2510:afr_sh_erase_pending] 0-gv3-replicate-0: Delta matrix for: bc60dc7ce87f0000 >[2013-03-07 17:00:46.983319] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.983347] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ] >[2013-03-07 17:00:46.983363] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ] >[2013-03-07 17:00:46.983390] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.983418] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.983462] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.983496] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.983513] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.983549] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.983580] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.983596] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.983632] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.983723] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.984020] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.984064] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.984098] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.984114] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.984153] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.984184] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.984200] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.984236] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.984267] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.984282] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.984320] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.984473] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.984696] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.984741] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.984758] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received >[2013-03-07 17:00:46.984771] T [afr-lk-common.c:1477:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: All servers locked. Calling the cbk >[2013-03-07 17:00:46.984787] D [afr-self-heal-data.c:1147:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by bc60dc7ce87f0000. Proceeding to self-heal >[2013-03-07 17:00:46.984802] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.984825] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.984841] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.984880] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.984909] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.984925] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.984962] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.984990] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.985006] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.985039] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.985089] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.985108] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.985145] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.985177] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.985194] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.985232] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.985263] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.985283] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.985315] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.985359] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.985433] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.985493] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.985515] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked >[2013-03-07 17:00:46.985565] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.985592] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded >[2013-03-07 17:00:46.985655] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.985685] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded >[2013-03-07 17:00:46.985759] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.985787] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded >[2013-03-07 17:00:46.985813] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.985828] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.985865] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.985894] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.985909] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.985943] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.985970] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.985985] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.986018] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.986084] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.986109] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded >[2013-03-07 17:00:46.986351] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.986374] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded >[2013-03-07 17:00:46.986399] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.986443] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded >[2013-03-07 17:00:46.986483] D [afr-self-heal-data.c:692:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: Pending matrix for: bc60dc7ce87f0000 >[2013-03-07 17:00:46.986504] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.986518] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.986531] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.986545] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 0 >[2013-03-07 17:00:46.986573] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.986590] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68 >[2013-03-07 17:00:46.986631] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.986663] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.986679] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68 >[2013-03-07 17:00:46.986714] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.986742] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.986758] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68 >[2013-03-07 17:00:46.986795] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.986943] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.987213] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.987250] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.987271] D [afr-self-heal-data.c:325:afr_sh_data_finish] 0-gv3-replicate-0: finishing data selfheal of <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.987286] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.987309] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.987325] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.987365] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.987394] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.987420] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.987459] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.987489] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.987514] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.987552] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.987665] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.987698] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.987713] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.987751] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.987796] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.987905] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.987950] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.987969] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked >[2013-03-07 17:00:46.987984] D [afr-self-heal-data.c:143:afr_sh_data_close] 0-gv3-replicate-0: closing fd of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 >[2013-03-07 17:00:46.988013] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.988029] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.988066] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.988087] D [afr-self-heal-data.c:143:afr_sh_data_close] 0-gv3-replicate-0: closing fd of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 >[2013-03-07 17:00:46.988110] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.988125] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.988162] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.988182] D [afr-self-heal-data.c:143:afr_sh_data_close] 0-gv3-replicate-0: closing fd of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 >[2013-03-07 17:00:46.988204] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000 >[2013-03-07 17:00:46.988219] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.988254] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.988294] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.988328] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.988344] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.988388] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.988434] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.988454] T [client-lk.c:391:delete_granted_locks_owner] 0-gv3-client-0: Number of locks cleared=0 >[2013-03-07 17:00:46.988473] T [client-rpc-fops.c:924:client3_3_flush_cbk] 0-gv3-client-0: deleting locks of owner (bc60dc7ce87f0000) returned 0 >[2013-03-07 17:00:46.988551] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.988572] T [client-lk.c:391:delete_granted_locks_owner] 0-gv3-client-2: Number of locks cleared=0 >[2013-03-07 17:00:46.988588] T [client-rpc-fops.c:924:client3_3_flush_cbk] 0-gv3-client-2: deleting locks of owner (bc60dc7ce87f0000) returned 0 >[2013-03-07 17:00:46.988616] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.988633] T [client-lk.c:391:delete_granted_locks_owner] 0-gv3-client-1: Number of locks cleared=0 >[2013-03-07 17:00:46.988649] T [client-rpc-fops.c:924:client3_3_flush_cbk] 0-gv3-client-1: deleting locks of owner (bc60dc7ce87f0000) returned 0 >[2013-03-07 17:00:46.988664] D [afr-self-heal-common.c:2216:afr_self_heal_completion_cbk] 0-gv3-replicate-0: background meta-data data self-heal completed on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.988688] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.988703] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.988716] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.988730] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.988744] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 2 >[2013-03-07 17:00:46.988756] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 2 for <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.988773] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 2 >[2013-03-07 17:00:46.988834] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-0: sending release on fd >[2013-03-07 17:00:46.988857] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.988874] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64 >[2013-03-07 17:00:46.988895] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.988914] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x56x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.988946] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 116, payload: 48, rpc hdr: 68 >[2013-03-07 17:00:46.988966] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-1: sending release on fd >[2013-03-07 17:00:46.988999] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.989006] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64 >[2013-03-07 17:00:46.989023] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.989043] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-2: sending release on fd >[2013-03-07 17:00:46.989016] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.989115] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64 >[2013-03-07 17:00:46.989149] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989192] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989213] D [afr-lk-common.c:1098:afr_lock_blocking] 0-gv3-replicate-0: we're done locking >[2013-03-07 17:00:46.989230] D [afr-self-heal-data.c:1112:afr_sh_data_post_blocking_inodelk_cbk] 0-gv3-replicate-0: Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by 7089dc7ce87f0000. Proceding to self-heal >[2013-03-07 17:00:46.989274] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.989292] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.989330] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x57x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.989362] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.989378] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.989426] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.989460] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.989476] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68 >[2013-03-07 17:00:46.989509] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989550] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x56x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.989581] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989627] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.989657] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989686] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x57x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.989694] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.989710] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded >[2013-03-07 17:00:46.989732] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.989812] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989837] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.989867] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded >[2013-03-07 17:00:46.989949] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.989974] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded >[2013-03-07 17:00:46.989999] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.990015] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.990054] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x58x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.990083] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.990099] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.990135] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.990163] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.990179] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68 >[2013-03-07 17:00:46.990215] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.990253] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.990293] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x58x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.990318] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded >[2013-03-07 17:00:46.990318] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-2 >[2013-03-07 17:00:46.990379] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.990381] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.990429] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded >[2013-03-07 17:00:46.990434] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.990447] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.990498] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded >[2013-03-07 17:00:46.990511] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.990518] D [afr-self-heal-data.c:692:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: Pending matrix for: 7089dc7ce87f0000 >[2013-03-07 17:00:46.990588] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.990603] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.990616] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.990630] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 0 >[2013-03-07 17:00:46.990646] D [afr-self-heal-data.c:738:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: No self-heal needed for <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.990660] D [afr-self-heal-data.c:325:afr_sh_data_finish] 0-gv3-replicate-0: finishing data selfheal of <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.990673] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal >[2013-03-07 17:00:46.990698] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.990715] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.990759] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x59x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.990789] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.990805] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.990840] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.990869] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000 >[2013-03-07 17:00:46.990885] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68 >[2013-03-07 17:00:46.990919] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.990957] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.991049] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x59x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.991074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.991119] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.991143] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.991197] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.991293] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.991316] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked >[2013-03-07 17:00:46.991331] D [afr-self-heal-common.c:2216:afr_self_heal_completion_cbk] 0-gv3-replicate-0: background meta-data data self-heal completed on <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.991354] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ] >[2013-03-07 17:00:46.991369] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.991382] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ] >[2013-03-07 17:00:46.991395] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2 >[2013-03-07 17:00:46.991416] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 1 >[2013-03-07 17:00:46.991431] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 1 for <gfid:78a027b5-dae3-4851-904c-926034438fb4> >[2013-03-07 17:00:46.991447] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 1 >[2013-03-07 17:00:46.991499] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-0: sending release on fd >[2013-03-07 17:00:46.991521] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.991550] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64 >[2013-03-07 17:00:46.991561] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.991568] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x60x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.991581] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 116, payload: 48, rpc hdr: 68 >[2013-03-07 17:00:46.991588] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-1: sending release on fd >[2013-03-07 17:00:46.991597] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.991603] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64 >[2013-03-07 17:00:46.991608] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.991628] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.991655] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-2: sending release on fd >[2013-03-07 17:00:46.991675] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: >[2013-03-07 17:00:46.991690] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64 >[2013-03-07 17:00:46.991728] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.991779] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.991824] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x60x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-0) >[2013-03-07 17:00:46.991839] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX 484a792f-dd0b-4c99-9d77-3cb20c96c621 >[2013-03-07 17:00:46.991905] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.991927] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.991993] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.992030] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.992067] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.992175] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.992162] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.992216] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.992269] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.992374] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.992403] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.992428] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.992482] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000 >[2013-03-07 17:00:46.992530] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.992528] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.992551] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-1 >[2013-03-07 17:00:46.992593] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.992611] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.992654] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68 >[2013-03-07 17:00:46.992690] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.992809] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2) >[2013-03-07 17:00:46.992894] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-2 >[2013-03-07 17:00:46.992958] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.993020] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.993059] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68 >[2013-03-07 17:00:46.993109] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.993368] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.993433] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX cfb7526b-cb8e-4c9a-90c1-0009a48b9d27 >[2013-03-07 17:00:46.993476] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.993494] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.993548] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.993848] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.993877] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0 >[2013-03-07 17:00:46.993891] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1 >[2013-03-07 17:00:46.993941] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000 >[2013-03-07 17:00:46.993965] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68 >[2013-03-07 17:00:46.994008] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.994250] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1) >[2013-03-07 17:00:46.994338] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-1 >[2013-03-07 17:00:49.885368] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-glusterfs: breaking reconnect chain >[2013-03-07 17:00:50.577967] T [socket.c:790:__socket_keepalive] 0-glusterfs: Keep-alive enabled for socket 15, interval 2, idle: 20 >[2013-03-07 17:00:50.885544] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv13-client-0: breaking reconnect chain >[2013-03-07 17:00:50.885587] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv13-client-1: breaking reconnect chain >[2013-03-07 17:00:50.885602] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv3-client-0: breaking reconnect chain >[2013-03-07 17:00:50.885614] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv14-client-1: breaking reconnect chain >[2013-03-07 17:00:50.885626] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv3-client-1: breaking reconnect chain >[2013-03-07 17:00:50.885638] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv3-client-2: breaking reconnect chain >[2013-03-07 17:00:50.885649] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv14-client-0: breaking reconnect chain
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 918437
:
706248
| 706770