Attachment 706770 Details for Bug 918437 – self-heal trace log

self-heal trace log

glustershd.log (text/plain), 273.33 KB, created by Todd Stansell on 2013-03-07 19:12:44 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Todd Stansell

Created: 2013-03-07 19:12:44 UTC

Size: 273.33 KB

patch

obsolete

>[2013-03-07 17:00:37.459049] E [afr-common.c:3761:afr_notify] 0-gv3-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2013-03-07 17:00:39.237462] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect
>[2013-03-07 17:00:39.237514] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:39.237535] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007
>[2013-03-07 17:00:39.237550] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:39.237562] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:39.243415] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:39.243457] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 7
>[2013-03-07 17:00:39.243476] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 7, interval 2, idle: 20
>[2013-03-07 17:00:39.243510] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:39.243582] E [socket.c:2767:socket_connect] 0-gv3-client-0: connection attempt failed (Connection refused)
>[2013-03-07 17:00:42.243906] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect
>[2013-03-07 17:00:42.243956] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:42.243976] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007
>[2013-03-07 17:00:42.243992] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:42.244005] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:42.249988] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:42.250039] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 7
>[2013-03-07 17:00:42.250058] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 7, interval 2, idle: 20
>[2013-03-07 17:00:42.250095] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:42.250168] E [socket.c:2767:socket_connect] 0-gv3-client-0: connection attempt failed (Connection refused)
>[2013-03-07 17:00:45.250523] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect
>[2013-03-07 17:00:45.255828] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-127.0.0.1 (port-24007) for hostname: localhost and port: 24007
>[2013-03-07 17:00:45.255872] T [socket.c:731:__socket_nodelay] 0-glusterfs: NODELAY enabled for socket 7
>[2013-03-07 17:00:45.255891] T [socket.c:790:__socket_keepalive] 0-glusterfs: Keep-alive enabled for socket 7, interval 2, idle: 20
>[2013-03-07 17:00:45.255928] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:45.256036] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect
>[2013-03-07 17:00:45.256073] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:45.256080] T [rpc-clnt.c:1301:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:45.256101] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007
>[2013-03-07 17:00:45.256120] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 32, rpc hdr: 64
>[2013-03-07 17:00:45.256125] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:45.256168] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:45.256204] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 2) to rpc-transport (glusterfs)
>[2013-03-07 17:00:45.256244] D [glusterfsd-mgmt.c:2152:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given
>[2013-03-07 17:00:45.262216] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:45.262269] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 8
>[2013-03-07 17:00:45.262290] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 8, interval 2, idle: 20
>[2013-03-07 17:00:45.262325] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:45.262469] D [client.c:2050:client_rpc_notify] 0-gv3-client-0: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:45.262506] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:45.262527] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:45.262596] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7f43f77e8f42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7f43f77cc0c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected
>[2013-03-07 17:00:45.262662] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x93x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:45.262689] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:45.611442] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 2) from rpc-transport (glusterfs)
>[2013-03-07 17:00:45.611500] I [glusterfsd-mgmt.c:1583:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2013-03-07 17:00:45.611539] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x93x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:45.611566] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-0: detected portmapper on server
>[2013-03-07 17:00:45.611594] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:45.611613] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64
>[2013-03-07 17:00:45.611688] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7f43f77e7762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7f43f77cc0c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected
>[2013-03-07 17:00:45.611749] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x94x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:45.611774] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:45.611876] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x94x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:45.611920] D [client-handshake.c:1741:client_query_portmap_cbk] 0-gv3-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2013-03-07 17:00:45.611978] D [socket.c:486:__socket_rwv] 0-gv3-client-0: EOF on socket
>[2013-03-07 17:00:45.612001] W [socket.c:514:__socket_rwv] 0-gv3-client-0: readv failed (No data available)
>[2013-03-07 17:00:45.612016] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-0: reading from socket failed. Error (No data available), peer (172.22.6.136:24007)
>[2013-03-07 17:00:45.612031] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:45.612059] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-0: cleaning up state in transport object 0xa4e010
>[2013-03-07 17:00:45.612079] I [client.c:2097:client_rpc_notify] 0-gv3-client-0: disconnected
>[2013-03-07 17:00:45.857663] W [glusterfsd.c:970:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x398f6e767d] (-->/lib64/libpthread.so.0() [0x398fa07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x40523d]))) 0-: received signum (15), shutting down
>[2013-03-07 17:00:45.857691] D [glusterfsd-mgmt.c:2214:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
>[2013-03-07 17:00:45.857721] T [glusterfsd.c:1489:glusterfs_pidfile_cleanup] 0-glusterfsd: pidfile /var/lib/glusterd/glustershd/run/glustershd.pid cleanup
>[2013-03-07 17:00:46.870036] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0alpha (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/876d8b0cbe47b5e0a51969bcdebaac27.socket --xlator-option *replicate*.node-uuid=860e1ecd-78c7-4a99-a642-e8eac0919a56)
>[2013-03-07 17:00:46.873936] I [socket.c:3459:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-03-07 17:00:46.873995] I [socket.c:3474:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-03-07 17:00:46.874182] I [socket.c:3459:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-03-07 17:00:46.874216] I [socket.c:3474:socket_init] 0-glusterfs: using system polling thread
>[2013-03-07 17:00:46.884836] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.886417] I [graph.c:239:gf_add_cmdline_options] 0-gv13-replicate-0: adding option 'node-uuid' for volume 'gv13-replicate-0' with value '860e1ecd-78c7-4a99-a642-e8eac0919a56'
>[2013-03-07 17:00:46.886429] I [graph.c:239:gf_add_cmdline_options] 0-gv3-replicate-0: adding option 'node-uuid' for volume 'gv3-replicate-0' with value '860e1ecd-78c7-4a99-a642-e8eac0919a56'
>[2013-03-07 17:00:46.886437] I [graph.c:239:gf_add_cmdline_options] 0-gv14-replicate-0: adding option 'node-uuid' for volume 'gv14-replicate-0' with value '860e1ecd-78c7-4a99-a642-e8eac0919a56'
>[2013-03-07 17:00:46.886492] T [options.c:1111:xlator_option_init_xlator] 0-gv13-replicate-0: option read-subvolume not set
>[2013-03-07 17:00:46.886508] T [options.c:1106:xlator_option_init_int32] 0-gv13-replicate-0: option read-subvolume-index using default value -1
>[2013-03-07 17:00:46.886515] T [options.c:77:xlator_option_validate_int] 0-gv13-replicate-0: no range check required for 'option read-subvolume-index -1'
>[2013-03-07 17:00:46.886521] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option choose-local using default value true
>[2013-03-07 17:00:46.886528] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option read-hash-mode using default value 0
>[2013-03-07 17:00:46.886535] T [options.c:1111:xlator_option_init_xlator] 0-gv13-replicate-0: option favorite-child not set
>[2013-03-07 17:00:46.886542] D [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option background-self-heal-count using set value 0
>[2013-03-07 17:00:46.886548] D [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option data-self-heal using set value on
>[2013-03-07 17:00:46.886555] T [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option data-self-heal-algorithm not set
>[2013-03-07 17:00:46.886561] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option data-self-heal-window-size using default value 1
>[2013-03-07 17:00:46.886567] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option metadata-self-heal using set value on
>[2013-03-07 17:00:46.886573] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option entry-self-heal using set value on
>[2013-03-07 17:00:46.886579] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option self-heal-daemon using set value on
>[2013-03-07 17:00:46.886586] D [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option iam-self-heal-daemon using set value yes
>[2013-03-07 17:00:46.886592] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option data-change-log using default value on
>[2013-03-07 17:00:46.886599] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option metadata-change-log using default value on
>[2013-03-07 17:00:46.886605] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option entry-change-log using default value on
>[2013-03-07 17:00:46.886611] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option optimistic-change-log using default value on
>[2013-03-07 17:00:46.886617] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option inodelk-trace using default value off
>[2013-03-07 17:00:46.886624] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option entrylk-trace using default value off
>[2013-03-07 17:00:46.886630] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option strict-readdir using default value off
>[2013-03-07 17:00:46.886636] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option eager-lock using default value on
>[2013-03-07 17:00:46.886642] T [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option quorum-type using default value none
>[2013-03-07 17:00:46.886649] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option quorum-count not set
>[2013-03-07 17:00:46.886660] T [options.c:1107:xlator_option_init_size] 0-gv13-replicate-0: option self-heal-readdir-size using default value 1KB
>[2013-03-07 17:00:46.886674] T [options.c:1105:xlator_option_init_uint32] 0-gv13-replicate-0: option post-op-delay-secs using default value 1
>[2013-03-07 17:00:46.886681] T [options.c:1110:xlator_option_init_bool] 0-gv13-replicate-0: option readdir-failover using default value on
>[2013-03-07 17:00:46.888320] D [options.c:1102:xlator_option_init_str] 0-gv13-replicate-0: option node-uuid using set value 860e1ecd-78c7-4a99-a642-e8eac0919a56
>[2013-03-07 17:00:46.888332] T [options.c:1106:xlator_option_init_int32] 0-gv13-replicate-0: option heal-timeout using default value 600
>[2013-03-07 17:00:46.888341] D [client.c:2331:client_init_grace_timer] 0-gv13-client-1: lk-heal = off
>[2013-03-07 17:00:46.888348] D [client.c:2342:client_init_grace_timer] 0-gv13-client-1: Client grace timeout value = 10
>[2013-03-07 17:00:46.888354] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-1: option frame-timeout using default value 1800
>[2013-03-07 17:00:46.888361] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-1: option remote-port not set
>[2013-03-07 17:00:46.888367] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-1: option ping-timeout using default value 42
>[2013-03-07 17:00:46.888373] D [options.c:1112:xlator_option_init_path] 0-gv13-client-1: option remote-subvolume using set value /gfs0/brick13
>[2013-03-07 17:00:46.888380] T [options.c:1110:xlator_option_init_bool] 0-gv13-client-1: option filter-O_DIRECT using default value disable
>[2013-03-07 17:00:46.888904] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv13-client-1: defaulting frame-timeout to 30mins
>[2013-03-07 17:00:46.888917] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so
>[2013-03-07 17:00:46.888935] I [socket.c:3459:socket_init] 0-gv13-client-1: SSL support is NOT enabled
>[2013-03-07 17:00:46.888942] I [socket.c:3474:socket_init] 0-gv13-client-1: using system polling thread
>[2013-03-07 17:00:46.888948] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv13-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-03-07 17:00:46.888957] D [client.c:2306:client_init_rpc] 0-gv13-client-1: client init successful
>[2013-03-07 17:00:46.888963] D [client.c:2331:client_init_grace_timer] 0-gv13-client-0: lk-heal = off
>[2013-03-07 17:00:46.888968] D [client.c:2342:client_init_grace_timer] 0-gv13-client-0: Client grace timeout value = 10
>[2013-03-07 17:00:46.888974] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-0: option frame-timeout using default value 1800
>[2013-03-07 17:00:46.888981] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-0: option remote-port not set
>[2013-03-07 17:00:46.888986] T [options.c:1106:xlator_option_init_int32] 0-gv13-client-0: option ping-timeout using default value 42
>[2013-03-07 17:00:46.888992] D [options.c:1112:xlator_option_init_path] 0-gv13-client-0: option remote-subvolume using set value /gfs0/brick13
>[2013-03-07 17:00:46.888999] T [options.c:1110:xlator_option_init_bool] 0-gv13-client-0: option filter-O_DIRECT using default value disable
>[2013-03-07 17:00:46.889539] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv13-client-0: defaulting frame-timeout to 30mins
>[2013-03-07 17:00:46.889551] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so
>[2013-03-07 17:00:46.889567] I [socket.c:3459:socket_init] 0-gv13-client-0: SSL support is NOT enabled
>[2013-03-07 17:00:46.889573] I [socket.c:3474:socket_init] 0-gv13-client-0: using system polling thread
>[2013-03-07 17:00:46.889579] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv13-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-03-07 17:00:46.889585] D [client.c:2306:client_init_rpc] 0-gv13-client-0: client init successful
>[2013-03-07 17:00:46.889596] T [options.c:1111:xlator_option_init_xlator] 0-gv3-replicate-0: option read-subvolume not set
>[2013-03-07 17:00:46.889603] T [options.c:1106:xlator_option_init_int32] 0-gv3-replicate-0: option read-subvolume-index using default value -1
>[2013-03-07 17:00:46.889609] T [options.c:77:xlator_option_validate_int] 0-gv3-replicate-0: no range check required for 'option read-subvolume-index -1'
>[2013-03-07 17:00:46.889615] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option choose-local using default value true
>[2013-03-07 17:00:46.889621] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option read-hash-mode using default value 0
>[2013-03-07 17:00:46.889628] T [options.c:1111:xlator_option_init_xlator] 0-gv3-replicate-0: option favorite-child not set
>[2013-03-07 17:00:46.889634] D [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option background-self-heal-count using set value 0
>[2013-03-07 17:00:46.889640] D [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option data-self-heal using set value on
>[2013-03-07 17:00:46.889647] T [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option data-self-heal-algorithm not set
>[2013-03-07 17:00:46.889653] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option data-self-heal-window-size using default value 1
>[2013-03-07 17:00:46.889660] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option metadata-self-heal using set value on
>[2013-03-07 17:00:46.889666] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option entry-self-heal using set value on
>[2013-03-07 17:00:46.889672] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option self-heal-daemon using set value on
>[2013-03-07 17:00:46.889678] D [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option iam-self-heal-daemon using set value yes
>[2013-03-07 17:00:46.889685] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option data-change-log using default value on
>[2013-03-07 17:00:46.889691] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option metadata-change-log using default value on
>[2013-03-07 17:00:46.889697] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option entry-change-log using default value on
>[2013-03-07 17:00:46.889703] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option optimistic-change-log using default value on
>[2013-03-07 17:00:46.889709] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option inodelk-trace using default value off
>[2013-03-07 17:00:46.889716] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option entrylk-trace using default value off
>[2013-03-07 17:00:46.889722] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option strict-readdir using default value off
>[2013-03-07 17:00:46.889728] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option eager-lock using default value on
>[2013-03-07 17:00:46.889734] T [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option quorum-type using default value none
>[2013-03-07 17:00:46.889741] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option quorum-count not set
>[2013-03-07 17:00:46.889747] T [options.c:1107:xlator_option_init_size] 0-gv3-replicate-0: option self-heal-readdir-size using default value 1KB
>[2013-03-07 17:00:46.889755] T [options.c:1105:xlator_option_init_uint32] 0-gv3-replicate-0: option post-op-delay-secs using default value 1
>[2013-03-07 17:00:46.889762] T [options.c:1110:xlator_option_init_bool] 0-gv3-replicate-0: option readdir-failover using default value on
>[2013-03-07 17:00:46.891526] D [options.c:1102:xlator_option_init_str] 0-gv3-replicate-0: option node-uuid using set value 860e1ecd-78c7-4a99-a642-e8eac0919a56
>[2013-03-07 17:00:46.891540] T [options.c:1106:xlator_option_init_int32] 0-gv3-replicate-0: option heal-timeout using default value 600
>[2013-03-07 17:00:46.891548] D [client.c:2331:client_init_grace_timer] 0-gv3-client-2: lk-heal = off
>[2013-03-07 17:00:46.891557] D [client.c:2342:client_init_grace_timer] 0-gv3-client-2: Client grace timeout value = 10
>[2013-03-07 17:00:46.891564] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-2: option frame-timeout using default value 1800
>[2013-03-07 17:00:46.891570] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-2: option remote-port not set
>[2013-03-07 17:00:46.891576] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-2: option ping-timeout using default value 42
>[2013-03-07 17:00:46.891582] D [options.c:1112:xlator_option_init_path] 0-gv3-client-2: option remote-subvolume using set value /gfs0/brick3
>[2013-03-07 17:00:46.891589] T [options.c:1110:xlator_option_init_bool] 0-gv3-client-2: option filter-O_DIRECT using default value disable
>[2013-03-07 17:00:46.892133] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv3-client-2: defaulting frame-timeout to 30mins
>[2013-03-07 17:00:46.892144] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so
>[2013-03-07 17:00:46.892161] I [socket.c:3459:socket_init] 0-gv3-client-2: SSL support is NOT enabled
>[2013-03-07 17:00:46.892167] I [socket.c:3474:socket_init] 0-gv3-client-2: using system polling thread
>[2013-03-07 17:00:46.892173] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv3-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-03-07 17:00:46.892179] D [client.c:2306:client_init_rpc] 0-gv3-client-2: client init successful
>[2013-03-07 17:00:46.892185] D [client.c:2331:client_init_grace_timer] 0-gv3-client-1: lk-heal = off
>[2013-03-07 17:00:46.892190] D [client.c:2342:client_init_grace_timer] 0-gv3-client-1: Client grace timeout value = 10
>[2013-03-07 17:00:46.892196] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-1: option frame-timeout using default value 1800
>[2013-03-07 17:00:46.892202] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-1: option remote-port not set
>[2013-03-07 17:00:46.892208] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-1: option ping-timeout using default value 42
>[2013-03-07 17:00:46.892214] D [options.c:1112:xlator_option_init_path] 0-gv3-client-1: option remote-subvolume using set value /gfs0/brick2
>[2013-03-07 17:00:46.892221] T [options.c:1110:xlator_option_init_bool] 0-gv3-client-1: option filter-O_DIRECT using default value disable
>[2013-03-07 17:00:46.892760] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv3-client-1: defaulting frame-timeout to 30mins
>[2013-03-07 17:00:46.892771] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so
>[2013-03-07 17:00:46.892786] I [socket.c:3459:socket_init] 0-gv3-client-1: SSL support is NOT enabled
>[2013-03-07 17:00:46.892795] I [socket.c:3474:socket_init] 0-gv3-client-1: using system polling thread
>[2013-03-07 17:00:46.892801] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv3-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-03-07 17:00:46.892807] D [client.c:2306:client_init_rpc] 0-gv3-client-1: client init successful
>[2013-03-07 17:00:46.892813] D [client.c:2331:client_init_grace_timer] 0-gv3-client-0: lk-heal = off
>[2013-03-07 17:00:46.892818] D [client.c:2342:client_init_grace_timer] 0-gv3-client-0: Client grace timeout value = 10
>[2013-03-07 17:00:46.892823] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-0: option frame-timeout using default value 1800
>[2013-03-07 17:00:46.892830] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-0: option remote-port not set
>[2013-03-07 17:00:46.892835] T [options.c:1106:xlator_option_init_int32] 0-gv3-client-0: option ping-timeout using default value 42
>[2013-03-07 17:00:46.892842] D [options.c:1112:xlator_option_init_path] 0-gv3-client-0: option remote-subvolume using set value /gfs0/brick1
>[2013-03-07 17:00:46.892848] T [options.c:1110:xlator_option_init_bool] 0-gv3-client-0: option filter-O_DIRECT using default value disable
>[2013-03-07 17:00:46.893363] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv3-client-0: defaulting frame-timeout to 30mins
>[2013-03-07 17:00:46.893377] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so
>[2013-03-07 17:00:46.893392] I [socket.c:3459:socket_init] 0-gv3-client-0: SSL support is NOT enabled
>[2013-03-07 17:00:46.893398] I [socket.c:3474:socket_init] 0-gv3-client-0: using system polling thread
>[2013-03-07 17:00:46.893404] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv3-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-03-07 17:00:46.893410] D [client.c:2306:client_init_rpc] 0-gv3-client-0: client init successful
>[2013-03-07 17:00:46.893417] T [options.c:1111:xlator_option_init_xlator] 0-gv14-replicate-0: option read-subvolume not set
>[2013-03-07 17:00:46.893423] T [options.c:1106:xlator_option_init_int32] 0-gv14-replicate-0: option read-subvolume-index using default value -1
>[2013-03-07 17:00:46.893429] T [options.c:77:xlator_option_validate_int] 0-gv14-replicate-0: no range check required for 'option read-subvolume-index -1'
>[2013-03-07 17:00:46.893435] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option choose-local using default value true
>[2013-03-07 17:00:46.893441] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option read-hash-mode using default value 0
>[2013-03-07 17:00:46.893448] T [options.c:1111:xlator_option_init_xlator] 0-gv14-replicate-0: option favorite-child not set
>[2013-03-07 17:00:46.893454] D [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option background-self-heal-count using set value 0
>[2013-03-07 17:00:46.893460] D [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option data-self-heal using set value on
>[2013-03-07 17:00:46.893467] T [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option data-self-heal-algorithm not set
>[2013-03-07 17:00:46.893473] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option data-self-heal-window-size using default value 1
>[2013-03-07 17:00:46.893480] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option metadata-self-heal using set value on
>[2013-03-07 17:00:46.893486] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option entry-self-heal using set value on
>[2013-03-07 17:00:46.893493] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option self-heal-daemon using set value on
>[2013-03-07 17:00:46.893499] D [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option iam-self-heal-daemon using set value yes
>[2013-03-07 17:00:46.893517] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option data-change-log using default value on
>[2013-03-07 17:00:46.893524] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option metadata-change-log using default value on
>[2013-03-07 17:00:46.893530] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option entry-change-log using default value on
>[2013-03-07 17:00:46.893537] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option optimistic-change-log using default value on
>[2013-03-07 17:00:46.893543] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option inodelk-trace using default value off
>[2013-03-07 17:00:46.893549] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option entrylk-trace using default value off
>[2013-03-07 17:00:46.893556] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option strict-readdir using default value off
>[2013-03-07 17:00:46.893562] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option eager-lock using default value on
>[2013-03-07 17:00:46.893568] T [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option quorum-type using default value none
>[2013-03-07 17:00:46.893575] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option quorum-count not set
>[2013-03-07 17:00:46.893581] T [options.c:1107:xlator_option_init_size] 0-gv14-replicate-0: option self-heal-readdir-size using default value 1KB
>[2013-03-07 17:00:46.893594] T [options.c:1105:xlator_option_init_uint32] 0-gv14-replicate-0: option post-op-delay-secs using default value 1
>[2013-03-07 17:00:46.893602] T [options.c:1110:xlator_option_init_bool] 0-gv14-replicate-0: option readdir-failover using default value on
>[2013-03-07 17:00:46.895341] D [options.c:1102:xlator_option_init_str] 0-gv14-replicate-0: option node-uuid using set value 860e1ecd-78c7-4a99-a642-e8eac0919a56
>[2013-03-07 17:00:46.895353] T [options.c:1106:xlator_option_init_int32] 0-gv14-replicate-0: option heal-timeout using default value 600
>[2013-03-07 17:00:46.895361] D [client.c:2331:client_init_grace_timer] 0-gv14-client-1: lk-heal = off
>[2013-03-07 17:00:46.895366] D [client.c:2342:client_init_grace_timer] 0-gv14-client-1: Client grace timeout value = 10
>[2013-03-07 17:00:46.895372] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-1: option frame-timeout using default value 1800
>[2013-03-07 17:00:46.895378] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-1: option remote-port not set
>[2013-03-07 17:00:46.895385] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-1: option ping-timeout using default value 42
>[2013-03-07 17:00:46.895391] D [options.c:1112:xlator_option_init_path] 0-gv14-client-1: option remote-subvolume using set value /gfs0/brick14
>[2013-03-07 17:00:46.895397] T [options.c:1110:xlator_option_init_bool] 0-gv14-client-1: option filter-O_DIRECT using default value disable
>[2013-03-07 17:00:46.895931] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv14-client-1: defaulting frame-timeout to 30mins
>[2013-03-07 17:00:46.895943] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so
>[2013-03-07 17:00:46.895959] I [socket.c:3459:socket_init] 0-gv14-client-1: SSL support is NOT enabled
>[2013-03-07 17:00:46.895966] I [socket.c:3474:socket_init] 0-gv14-client-1: using system polling thread
>[2013-03-07 17:00:46.895972] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv14-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-03-07 17:00:46.895977] D [client.c:2306:client_init_rpc] 0-gv14-client-1: client init successful
>[2013-03-07 17:00:46.895983] D [client.c:2331:client_init_grace_timer] 0-gv14-client-0: lk-heal = off
>[2013-03-07 17:00:46.895989] D [client.c:2342:client_init_grace_timer] 0-gv14-client-0: Client grace timeout value = 10
>[2013-03-07 17:00:46.895994] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-0: option frame-timeout using default value 1800
>[2013-03-07 17:00:46.896001] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-0: option remote-port not set
>[2013-03-07 17:00:46.896006] T [options.c:1106:xlator_option_init_int32] 0-gv14-client-0: option ping-timeout using default value 42
>[2013-03-07 17:00:46.896013] D [options.c:1112:xlator_option_init_path] 0-gv14-client-0: option remote-subvolume using set value /gfs0/brick14
>[2013-03-07 17:00:46.896019] T [options.c:1110:xlator_option_init_bool] 0-gv14-client-0: option filter-O_DIRECT using default value disable
>[2013-03-07 17:00:46.896551] D [rpc-clnt.c:964:rpc_clnt_connection_init] 0-gv14-client-0: defaulting frame-timeout to 30mins
>[2013-03-07 17:00:46.896563] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.0alpha/rpc-transport/socket.so
>[2013-03-07 17:00:46.896580] I [socket.c:3459:socket_init] 0-gv14-client-0: SSL support is NOT enabled
>[2013-03-07 17:00:46.896587] I [socket.c:3474:socket_init] 0-gv14-client-0: using system polling thread
>[2013-03-07 17:00:46.896593] D [rpc-clnt.c:1372:rpcclnt_cbk_program_register] 0-gv14-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
>[2013-03-07 17:00:46.896598] D [client.c:2306:client_init_rpc] 0-gv14-client-0: client init successful
>[2013-03-07 17:00:46.896625] I [client.c:2154:notify] 0-gv13-client-0: parent translators are ready, attempting connect on transport
>[2013-03-07 17:00:46.896631] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-0: attempting reconnect
>[2013-03-07 17:00:46.896643] D [name.c:155:client_fill_address_family] 0-gv13-client-0: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.896649] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-0: option remote-port missing in volume gv13-client-0. Defaulting to 24007
>[2013-03-07 17:00:46.896656] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.902895] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.902955] T [socket.c:731:__socket_nodelay] 0-gv13-client-0: NODELAY enabled for socket 10
>[2013-03-07 17:00:46.902976] T [socket.c:790:__socket_keepalive] 0-gv13-client-0: Keep-alive enabled for socket 10, interval 2, idle: 20
>[2013-03-07 17:00:46.903023] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.903170] I [client.c:2154:notify] 0-gv13-client-1: parent translators are ready, attempting connect on transport
>[2013-03-07 17:00:46.903195] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-1: attempting reconnect
>[2013-03-07 17:00:46.903214] D [name.c:155:client_fill_address_family] 0-gv13-client-1: address-family not specified, guessing it to be inet from (remote-host: host14)
>[2013-03-07 17:00:46.903229] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-1: option remote-port missing in volume gv13-client-1. Defaulting to 24007
>[2013-03-07 17:00:46.903244] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14
>[2013-03-07 17:00:46.925853] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007
>[2013-03-07 17:00:46.925869] T [socket.c:731:__socket_nodelay] 0-gv13-client-1: NODELAY enabled for socket 11
>[2013-03-07 17:00:46.925875] T [socket.c:790:__socket_keepalive] 0-gv13-client-1: Keep-alive enabled for socket 11, interval 2, idle: 20
>[2013-03-07 17:00:46.925887] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.926099] I [client.c:2154:notify] 0-gv3-client-0: parent translators are ready, attempting connect on transport
>[2013-03-07 17:00:46.926108] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect
>[2013-03-07 17:00:46.926115] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.926120] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007
>[2013-03-07 17:00:46.926124] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.928253] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.928266] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 12
>[2013-03-07 17:00:46.928272] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 12, interval 2, idle: 20
>[2013-03-07 17:00:46.928282] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.928316] I [client.c:2154:notify] 0-gv3-client-1: parent translators are ready, attempting connect on transport
>[2013-03-07 17:00:46.928337] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-1: attempting reconnect
>[2013-03-07 17:00:46.928344] D [name.c:155:client_fill_address_family] 0-gv3-client-1: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.928349] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-1: option remote-port missing in volume gv3-client-1. Defaulting to 24007
>[2013-03-07 17:00:46.928359] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.930513] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.930530] T [socket.c:731:__socket_nodelay] 0-gv3-client-1: NODELAY enabled for socket 13
>[2013-03-07 17:00:46.930536] T [socket.c:790:__socket_keepalive] 0-gv3-client-1: Keep-alive enabled for socket 13, interval 2, idle: 20
>[2013-03-07 17:00:46.930547] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.930583] I [client.c:2154:notify] 0-gv3-client-2: parent translators are ready, attempting connect on transport
>[2013-03-07 17:00:46.930603] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-2: attempting reconnect
>[2013-03-07 17:00:46.930610] D [name.c:155:client_fill_address_family] 0-gv3-client-2: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.930615] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-2: option remote-port missing in volume gv3-client-2. Defaulting to 24007
>[2013-03-07 17:00:46.930620] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.932807] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.932820] T [socket.c:731:__socket_nodelay] 0-gv3-client-2: NODELAY enabled for socket 14
>[2013-03-07 17:00:46.932826] T [socket.c:790:__socket_keepalive] 0-gv3-client-2: Keep-alive enabled for socket 14, interval 2, idle: 20
>[2013-03-07 17:00:46.932836] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.932883] I [client.c:2154:notify] 0-gv14-client-0: parent translators are ready, attempting connect on transport
>[2013-03-07 17:00:46.932892] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-0: attempting reconnect
>[2013-03-07 17:00:46.932899] D [name.c:155:client_fill_address_family] 0-gv14-client-0: address-family not specified, guessing it to be inet from (remote-host: host14)
>[2013-03-07 17:00:46.932904] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-0: option remote-port missing in volume gv14-client-0. Defaulting to 24007
>[2013-03-07 17:00:46.932908] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14
>[2013-03-07 17:00:46.935058] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007
>[2013-03-07 17:00:46.935071] T [socket.c:731:__socket_nodelay] 0-gv14-client-0: NODELAY enabled for socket 15
>[2013-03-07 17:00:46.935076] T [socket.c:790:__socket_keepalive] 0-gv14-client-0: Keep-alive enabled for socket 15, interval 2, idle: 20
>[2013-03-07 17:00:46.935087] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.935372] I [client.c:2154:notify] 0-gv14-client-1: parent translators are ready, attempting connect on transport
>[2013-03-07 17:00:46.935421] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-1: attempting reconnect
>[2013-03-07 17:00:46.935442] D [name.c:155:client_fill_address_family] 0-gv14-client-1: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.935458] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-1: option remote-port missing in volume gv14-client-1. Defaulting to 24007
>[2013-03-07 17:00:46.935473] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.937880] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.937900] T [socket.c:731:__socket_nodelay] 0-gv14-client-1: NODELAY enabled for socket 16
>[2013-03-07 17:00:46.937911] T [socket.c:790:__socket_keepalive] 0-gv14-client-1: Keep-alive enabled for socket 16, interval 2, idle: 20
>[2013-03-07 17:00:46.937924] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume gv14-client-0
>  2:     type protocol/client
>  3:     option password f2c735fd-01d5-416c-b7cc-643565a19e30
>  4:     option username 98840dc8-2d17-4c79-b9a0-62700d79daa9
>  5:     option transport-type tcp
>  6:     option remote-subvolume /gfs0/brick14
>  7:     option remote-host host14
>  8: end-volume
>  9: 
> 10: volume gv14-client-1
> 11:     type protocol/client
> 12:     option password f2c735fd-01d5-416c-b7cc-643565a19e30
> 13:     option username 98840dc8-2d17-4c79-b9a0-62700d79daa9
> 14:     option transport-type tcp
> 15:     option remote-subvolume /gfs0/brick14
> 16:     option remote-host host13
> 17: end-volume
> 18: 
> 19: volume gv14-replicate-0
> 20:     type cluster/replicate
> 21:     option iam-self-heal-daemon yes
> 22:     option self-heal-daemon on
> 23:     option entry-self-heal on
> 24:     option data-self-heal on
> 25:     option metadata-self-heal on
> 26:     option background-self-heal-count 0
> 27:     subvolumes gv14-client-0 gv14-client-1
> 28: end-volume
> 29: 
> 30: volume gv3-client-0
> 31:     type protocol/client
> 32:     option password 85e393d6-2d14-4c47-801f-0d01436dc457
> 33:     option username f29c81ad-73cb-49b4-a0ea-2dcfda202b4b
> 34:     option transport-type tcp
> 35:     option remote-subvolume /gfs0/brick1
> 36:     option remote-host host13
> 37: end-volume
> 38: 
> 39: volume gv3-client-1
> 40:     type protocol/client
> 41:     option password 85e393d6-2d14-4c47-801f-0d01436dc457
> 42:     option username f29c81ad-73cb-49b4-a0ea-2dcfda202b4b
> 43:     option transport-type tcp
> 44:     option remote-subvolume /gfs0/brick2
> 45:     option remote-host host13
> 46: end-volume
> 47: 
> 48: volume gv3-client-2
> 49:     type protocol/client
> 50:     option password 85e393d6-2d14-4c47-801f-0d01436dc457
> 51:     option username f29c81ad-73cb-49b4-a0ea-2dcfda202b4b
> 52:     option transport-type tcp
> 53:     option remote-subvolume /gfs0/brick3
> 54:     option remote-host host13
> 55: end-volume
> 56: 
> 57: volume gv3-replicate-0
> 58:     type cluster/replicate
> 59:     option iam-self-heal-daemon yes
> 60:     option self-heal-daemon on
> 61:     option entry-self-heal on
> 62:     option data-self-heal on
> 63:     option metadata-self-heal on
> 64:     option background-self-heal-count 0
> 65:     subvolumes gv3-client-0 gv3-client-1 gv3-client-2
> 66: end-volume
> 67: 
> 68: volume gv13-client-0
> 69:     type protocol/client
> 70:     option password a5bb356d-e400-47b5-99da-dc91b78929b1
> 71:     option username 305f0d14-54f0-4d35-991c-e76d89f4b307
> 72:     option transport-type tcp
> 73:     option remote-subvolume /gfs0/brick13
> 74:     option remote-host host13
> 75: end-volume
> 76: 
> 77: volume gv13-client-1
> 78:     type protocol/client
> 79:     option password a5bb356d-e400-47b5-99da-dc91b78929b1
> 80:     option username 305f0d14-54f0-4d35-991c-e76d89f4b307
> 81:     option transport-type tcp
> 82:     option remote-subvolume /gfs0/brick13
> 83:     option remote-host host14
> 84: end-volume
> 85: 
> 86: volume gv13-replicate-0
> 87:     type cluster/replicate
> 88:     option iam-self-heal-daemon yes
> 89:     option self-heal-daemon on
> 90:     option entry-self-heal on
> 91:     option data-self-heal on
> 92:     option metadata-self-heal on
> 93:     option background-self-heal-count 0
> 94:     subvolumes gv13-client-0 gv13-client-1
> 95: end-volume
> 96: 
> 97: volume glustershd
> 98:     type debug/io-stats
> 99:     option log-level TRACE
>100:     subvolumes gv13-replicate-0 gv3-replicate-0 gv14-replicate-0
>101: end-volume
>
>+------------------------------------------------------------------------------+
>[2013-03-07 17:00:46.938045] D [glusterfsd-mgmt.c:2152:glusterfs_mgmt_pmap_signin] 0-fsd-mgmt: portmapper signin arguments not given
>[2013-03-07 17:00:46.938077] D [client.c:2050:client_rpc_notify] 0-gv13-client-0: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.938104] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938115] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.938265] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.938289] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.938302] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938315] D [client.c:2050:client_rpc_notify] 0-gv13-client-1: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.938323] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938329] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.938347] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.938360] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.938367] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938377] D [client.c:2050:client_rpc_notify] 0-gv3-client-0: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.938391] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938408] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.938425] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.938441] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.938448] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938469] D [client.c:2050:client_rpc_notify] 0-gv3-client-1: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.938477] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938482] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.938498] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.938518] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.938529] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938539] D [client.c:2050:client_rpc_notify] 0-gv3-client-2: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.938547] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938553] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.938568] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected
>[2013-03-07 17:00:46.938594] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.938608] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938618] D [client.c:2050:client_rpc_notify] 0-gv14-client-0: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.938624] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938633] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.938660] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.938672] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.938678] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938688] D [client.c:2050:client_rpc_notify] 0-gv14-client-1: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.938697] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938702] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.938717] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.938743] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.938751] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938768] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.938780] D [client-handshake.c:1692:server_has_portmap] 0-gv13-client-0: detected portmapper on server
>[2013-03-07 17:00:46.938790] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938810] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64
>[2013-03-07 17:00:46.938829] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.938841] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.938848] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938860] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.938868] D [client-handshake.c:1692:server_has_portmap] 0-gv13-client-1: detected portmapper on server
>[2013-03-07 17:00:46.938876] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938881] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64
>[2013-03-07 17:00:46.938897] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.938918] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.938925] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.938935] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.938941] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-0: detected portmapper on server
>[2013-03-07 17:00:46.938949] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.938954] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64
>[2013-03-07 17:00:46.938968] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.938980] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.938998] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.939009] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.939016] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-1: detected portmapper on server
>[2013-03-07 17:00:46.939023] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.939032] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64
>[2013-03-07 17:00:46.939047] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.939060] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.939077] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.939087] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.939094] D [client-handshake.c:1692:server_has_portmap] 0-gv3-client-2: detected portmapper on server
>[2013-03-07 17:00:46.939101] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.939105] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 80, payload: 16, rpc hdr: 64
>[2013-03-07 17:00:46.939119] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected
>[2013-03-07 17:00:46.939149] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.939156] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.939167] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.939174] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv13-client-0: changing port to 49152 (from 0)
>[2013-03-07 17:00:46.939190] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.939196] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv13-client-1: changing port to 49152 (from 0)
>[2013-03-07 17:00:46.939206] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.939212] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv3-client-0: changing port to 49154 (from 0)
>[2013-03-07 17:00:46.939226] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.939234] D [client-handshake.c:1692:server_has_portmap] 0-gv14-client-1: detected portmapper on server
>[2013-03-07 17:00:46.939242] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.939247] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64
>[2013-03-07 17:00:46.939262] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.939274] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.939284] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.939295] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x1x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.939302] D [client-handshake.c:1692:server_has_portmap] 0-gv14-client-0: detected portmapper on server
>[2013-03-07 17:00:46.939309] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.939315] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 84, payload: 20, rpc hdr: 64
>[2013-03-07 17:00:46.939341] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_query_portmap+0x272) [0x7fe879bde762] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.939351] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.939357] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.939365] D [socket.c:486:__socket_rwv] 0-gv13-client-0: EOF on socket
>[2013-03-07 17:00:46.939372] W [socket.c:514:__socket_rwv] 0-gv13-client-0: readv failed (No data available)
>[2013-03-07 17:00:46.939377] D [socket.c:1962:__socket_proto_state_machine] 0-gv13-client-0: reading from socket failed. Error (No data available), peer (172.22.6.136:24007)
>[2013-03-07 17:00:46.939405] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:46.939416] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv13-client-0: cleaning up state in transport object 0x9d2140
>[2013-03-07 17:00:46.939423] D [client.c:2115:client_rpc_notify] 0-gv13-client-0: disconnected (skipped notify)
>[2013-03-07 17:00:46.939427] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-0: attempting reconnect
>[2013-03-07 17:00:46.939432] D [name.c:155:client_fill_address_family] 0-gv13-client-0: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.939437] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-0: option remote-port missing in volume gv13-client-0. Defaulting to 24007
>[2013-03-07 17:00:46.939442] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:46.939446] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.941713] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.941730] T [socket.c:731:__socket_nodelay] 0-gv13-client-0: NODELAY enabled for socket 7
>[2013-03-07 17:00:46.941737] T [socket.c:790:__socket_keepalive] 0-gv13-client-0: Keep-alive enabled for socket 7, interval 2, idle: 20
>[2013-03-07 17:00:46.941749] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.941793] D [socket.c:486:__socket_rwv] 0-gv13-client-1: EOF on socket
>[2013-03-07 17:00:46.941805] W [socket.c:514:__socket_rwv] 0-gv13-client-1: readv failed (No data available)
>[2013-03-07 17:00:46.941813] D [socket.c:1962:__socket_proto_state_machine] 0-gv13-client-1: reading from socket failed. Error (No data available), peer (172.22.6.137:24007)
>[2013-03-07 17:00:46.941826] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:46.941837] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv13-client-1: cleaning up state in transport object 0x9ad8d0
>[2013-03-07 17:00:46.941844] D [client.c:2115:client_rpc_notify] 0-gv13-client-1: disconnected (skipped notify)
>[2013-03-07 17:00:46.941848] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv13-client-1: attempting reconnect
>[2013-03-07 17:00:46.941853] D [name.c:155:client_fill_address_family] 0-gv13-client-1: address-family not specified, guessing it to be inet from (remote-host: host14)
>[2013-03-07 17:00:46.941858] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv13-client-1: option remote-port missing in volume gv13-client-1. Defaulting to 24007
>[2013-03-07 17:00:46.941863] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:46.941867] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14
>[2013-03-07 17:00:46.944056] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007
>[2013-03-07 17:00:46.944069] T [socket.c:731:__socket_nodelay] 0-gv13-client-1: NODELAY enabled for socket 9
>[2013-03-07 17:00:46.944075] T [socket.c:790:__socket_keepalive] 0-gv13-client-1: Keep-alive enabled for socket 9, interval 2, idle: 20
>[2013-03-07 17:00:46.944086] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.944359] D [socket.c:486:__socket_rwv] 0-gv3-client-0: EOF on socket
>[2013-03-07 17:00:46.944407] W [socket.c:514:__socket_rwv] 0-gv3-client-0: readv failed (No data available)
>[2013-03-07 17:00:46.944425] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-0: reading from socket failed. Error (No data available), peer (172.22.6.136:24007)
>[2013-03-07 17:00:46.944442] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:46.944481] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-0: cleaning up state in transport object 0xa61220
>[2013-03-07 17:00:46.944529] D [client.c:2115:client_rpc_notify] 0-gv3-client-0: disconnected (skipped notify)
>[2013-03-07 17:00:46.944546] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-0: attempting reconnect
>[2013-03-07 17:00:46.944563] D [name.c:155:client_fill_address_family] 0-gv3-client-0: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.944578] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-0: option remote-port missing in volume gv3-client-0. Defaulting to 24007
>[2013-03-07 17:00:46.944592] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:46.944605] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.947103] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.947123] T [socket.c:731:__socket_nodelay] 0-gv3-client-0: NODELAY enabled for socket 10
>[2013-03-07 17:00:46.947130] T [socket.c:790:__socket_keepalive] 0-gv3-client-0: Keep-alive enabled for socket 10, interval 2, idle: 20
>[2013-03-07 17:00:46.947142] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.947193] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.947202] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv14-client-1: changing port to 49153 (from 0)
>[2013-03-07 17:00:46.947220] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.947227] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv3-client-1: changing port to 49155 (from 0)
>[2013-03-07 17:00:46.947246] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.947253] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv3-client-2: changing port to 49156 (from 0)
>[2013-03-07 17:00:46.947266] D [socket.c:486:__socket_rwv] 0-gv14-client-1: EOF on socket
>[2013-03-07 17:00:46.947273] W [socket.c:514:__socket_rwv] 0-gv14-client-1: readv failed (No data available)
>[2013-03-07 17:00:46.947278] D [socket.c:1962:__socket_proto_state_machine] 0-gv14-client-1: reading from socket failed. Error (No data available), peer (172.22.6.136:24007)
>[2013-03-07 17:00:46.947283] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:46.947292] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv14-client-1: cleaning up state in transport object 0xaa7330
>[2013-03-07 17:00:46.947299] D [client.c:2115:client_rpc_notify] 0-gv14-client-1: disconnected (skipped notify)
>[2013-03-07 17:00:46.947304] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-1: attempting reconnect
>[2013-03-07 17:00:46.947310] D [name.c:155:client_fill_address_family] 0-gv14-client-1: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.947314] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-1: option remote-port missing in volume gv14-client-1. Defaulting to 24007
>[2013-03-07 17:00:46.947319] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:46.947323] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.949575] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.949593] T [socket.c:731:__socket_nodelay] 0-gv14-client-1: NODELAY enabled for socket 11
>[2013-03-07 17:00:46.949599] T [socket.c:790:__socket_keepalive] 0-gv14-client-1: Keep-alive enabled for socket 11, interval 2, idle: 20
>[2013-03-07 17:00:46.949611] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.949655] D [socket.c:486:__socket_rwv] 0-gv3-client-1: EOF on socket
>[2013-03-07 17:00:46.949664] W [socket.c:514:__socket_rwv] 0-gv3-client-1: readv failed (No data available)
>[2013-03-07 17:00:46.949669] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-1: reading from socket failed. Error (No data available), peer (172.22.6.136:24007)
>[2013-03-07 17:00:46.949673] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:46.949682] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-1: cleaning up state in transport object 0xa3ca30
>[2013-03-07 17:00:46.949689] D [client.c:2115:client_rpc_notify] 0-gv3-client-1: disconnected (skipped notify)
>[2013-03-07 17:00:46.949694] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-1: attempting reconnect
>[2013-03-07 17:00:46.949699] D [name.c:155:client_fill_address_family] 0-gv3-client-1: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.949704] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-1: option remote-port missing in volume gv3-client-1. Defaulting to 24007
>[2013-03-07 17:00:46.949708] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:46.949712] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.951924] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.951937] T [socket.c:731:__socket_nodelay] 0-gv3-client-1: NODELAY enabled for socket 12
>[2013-03-07 17:00:46.951943] T [socket.c:790:__socket_keepalive] 0-gv3-client-1: Keep-alive enabled for socket 12, interval 2, idle: 20
>[2013-03-07 17:00:46.951953] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.951997] D [socket.c:486:__socket_rwv] 0-gv3-client-2: EOF on socket
>[2013-03-07 17:00:46.952005] W [socket.c:514:__socket_rwv] 0-gv3-client-2: readv failed (No data available)
>[2013-03-07 17:00:46.952009] D [socket.c:1962:__socket_proto_state_machine] 0-gv3-client-2: reading from socket failed. Error (No data available), peer (172.22.6.136:24007)
>[2013-03-07 17:00:46.952014] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:46.952023] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv3-client-2: cleaning up state in transport object 0xa181e0
>[2013-03-07 17:00:46.952029] D [client.c:2115:client_rpc_notify] 0-gv3-client-2: disconnected (skipped notify)
>[2013-03-07 17:00:46.952034] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv3-client-2: attempting reconnect
>[2013-03-07 17:00:46.952039] D [name.c:155:client_fill_address_family] 0-gv3-client-2: address-family not specified, guessing it to be inet from (remote-host: host13)
>[2013-03-07 17:00:46.952043] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv3-client-2: option remote-port missing in volume gv3-client-2. Defaulting to 24007
>[2013-03-07 17:00:46.952048] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:46.952052] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host13
>[2013-03-07 17:00:46.954227] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.136 (port-24007) for hostname: host13 and port: 24007
>[2013-03-07 17:00:46.954241] T [socket.c:731:__socket_nodelay] 0-gv3-client-2: NODELAY enabled for socket 13
>[2013-03-07 17:00:46.954247] T [socket.c:790:__socket_keepalive] 0-gv3-client-2: Keep-alive enabled for socket 13, interval 2, idle: 20
>[2013-03-07 17:00:46.954258] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.954301] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x2x Program: Gluster Portmap, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.954310] I [rpc-clnt.c:1648:rpc_clnt_reconfig] 0-gv14-client-0: changing port to 49153 (from 0)
>[2013-03-07 17:00:46.954323] D [client.c:2050:client_rpc_notify] 0-gv13-client-0: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.954331] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.954337] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.954358] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.954379] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.954391] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.954408] D [client.c:2050:client_rpc_notify] 0-gv13-client-1: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.954418] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.954427] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.954454] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.954470] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.954484] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.954494] D [client.c:2050:client_rpc_notify] 0-gv3-client-0: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.954501] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.954507] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.954524] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.954543] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.954550] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.954567] D [socket.c:486:__socket_rwv] 0-gv14-client-0: EOF on socket
>[2013-03-07 17:00:46.954573] W [socket.c:514:__socket_rwv] 0-gv14-client-0: readv failed (No data available)
>[2013-03-07 17:00:46.954578] D [socket.c:1962:__socket_proto_state_machine] 0-gv14-client-0: reading from socket failed. Error (No data available), peer (172.22.6.137:24007)
>[2013-03-07 17:00:46.954582] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-03-07 17:00:46.954593] T [rpc-clnt.c:519:rpc_clnt_connection_cleanup] 0-gv14-client-0: cleaning up state in transport object 0xacbb80
>[2013-03-07 17:00:46.954599] D [client.c:2115:client_rpc_notify] 0-gv14-client-0: disconnected (skipped notify)
>[2013-03-07 17:00:46.954603] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-gv14-client-0: attempting reconnect
>[2013-03-07 17:00:46.954608] D [name.c:155:client_fill_address_family] 0-gv14-client-0: address-family not specified, guessing it to be inet from (remote-host: host14)
>[2013-03-07 17:00:46.954613] T [name.c:225:af_inet_client_get_remote_sockaddr] 0-gv14-client-0: option remote-port missing in volume gv14-client-0. Defaulting to 24007
>[2013-03-07 17:00:46.954618] T [common-utils.c:158:gf_resolve_ip6] 0-resolver: flushing DNS cache
>[2013-03-07 17:00:46.954622] T [common-utils.c:165:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: host14
>[2013-03-07 17:00:46.956851] D [common-utils.c:207:gf_resolve_ip6] 0-resolver: returning ip-172.22.6.137 (port-24007) for hostname: host14 and port: 24007
>[2013-03-07 17:00:46.956866] T [socket.c:731:__socket_nodelay] 0-gv14-client-0: NODELAY enabled for socket 14
>[2013-03-07 17:00:46.956872] T [socket.c:790:__socket_keepalive] 0-gv14-client-0: Keep-alive enabled for socket 14, interval 2, idle: 20
>[2013-03-07 17:00:46.956883] W [common-utils.c:2300:gf_ports_reserved] 0-glusterfs-socket:  is not a valid port identifier
>[2013-03-07 17:00:46.957166] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.957217] T [client-handshake.c:1664:select_server_supported_programs] 0-gv13-client-0: GF-DUMP (1) not supported
>[2013-03-07 17:00:46.957234] I [client-handshake.c:1658:select_server_supported_programs] 0-gv13-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-03-07 17:00:46.957293] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.957320] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64
>[2013-03-07 17:00:46.957403] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.957452] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.957476] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.957510] D [client.c:2050:client_rpc_notify] 0-gv14-client-1: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.957531] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.957548] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.957601] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.957684] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.957708] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.957738] D [client.c:2050:client_rpc_notify] 0-gv3-client-1: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.957757] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.957774] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.957821] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.957870] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.957891] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.957920] D [client.c:2050:client_rpc_notify] 0-gv3-client-2: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.957939] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.957955] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.957998] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected
>[2013-03-07 17:00:46.958050] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.958085] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.958153] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.958186] T [client-handshake.c:1664:select_server_supported_programs] 0-gv14-client-1: GF-DUMP (1) not supported
>[2013-03-07 17:00:46.958209] I [client-handshake.c:1658:select_server_supported_programs] 0-gv14-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-03-07 17:00:46.958267] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.958293] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64
>[2013-03-07 17:00:46.958365] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.958425] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.958455] D [client-handshake.c:185:client_start_ping] 0-gv14-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.958508] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.958531] T [client-handshake.c:1664:select_server_supported_programs] 0-gv13-client-1: GF-DUMP (1) not supported
>[2013-03-07 17:00:46.958544] I [client-handshake.c:1658:select_server_supported_programs] 0-gv13-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-03-07 17:00:46.958581] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.958599] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64
>[2013-03-07 17:00:46.958658] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.958690] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.958710] D [client-handshake.c:185:client_start_ping] 0-gv13-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.958742] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.958763] T [client-handshake.c:1664:select_server_supported_programs] 0-gv3-client-0: GF-DUMP (1) not supported
>[2013-03-07 17:00:46.958776] I [client-handshake.c:1658:select_server_supported_programs] 0-gv3-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-03-07 17:00:46.958810] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.958827] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64
>[2013-03-07 17:00:46.958871] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.958912] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.958960] D [client-handshake.c:185:client_start_ping] 0-gv3-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.959005] D [client.c:2050:client_rpc_notify] 0-gv14-client-0: got RPC_CLNT_CONNECT
>[2013-03-07 17:00:46.959034] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.959060] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64
>[2013-03-07 17:00:46.959123] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_handshake+0x182) [0x7fe879bdff42] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.959173] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.959201] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.959256] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.959345] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv13-client-1: clnt-lk-version = 1, server-lk-version = 0
>[2013-03-07 17:00:46.959376] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv13-client-1: Connected to 172.22.6.137:49152, attached to remote volume '/gfs0/brick13'.
>[2013-03-07 17:00:46.959425] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv13-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-03-07 17:00:46.959454] D [client-handshake.c:1318:client_post_handshake] 0-gv13-client-1: No fds to open - notifying all parents child up
>[2013-03-07 17:00:46.959481] D [client-handshake.c:486:client_set_lk_version] 0-gv13-client-1: Sending SET_LK_VERSION
>[2013-03-07 17:00:46.959512] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.959542] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64
>[2013-03-07 17:00:46.959587] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.959643] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.959671] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-03-07 17:00:46.959713] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.959747] I [afr-common.c:3724:afr_notify] 0-gv13-replicate-0: Subvolume 'gv13-client-1' came back up; going online.
>[2013-03-07 17:00:46.959874] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.959921] T [client-handshake.c:1664:select_server_supported_programs] 0-gv3-client-1: GF-DUMP (1) not supported
>[2013-03-07 17:00:46.959930] I [client-handshake.c:1658:select_server_supported_programs] 0-gv3-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-03-07 17:00:46.959961] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.959997] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64
>[2013-03-07 17:00:46.959994] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.960062] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.960078] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.960113] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.960140] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.960175] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.960222] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.960263] T [client-handshake.c:1664:select_server_supported_programs] 0-gv3-client-2: GF-DUMP (1) not supported
>[2013-03-07 17:00:46.960293] I [client-handshake.c:1658:select_server_supported_programs] 0-gv3-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-03-07 17:00:46.960350] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.960387] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64
>[2013-03-07 17:00:46.960459] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-2: connect () called on transport already connected
>[2013-03-07 17:00:46.960517] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.960551] D [client-handshake.c:185:client_start_ping] 0-gv3-client-2: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.960609] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.960656] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv13-client-1: Server lk version = 1
>[2013-03-07 17:00:46.960697] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x3x Program: GF-DUMP, ProgVers: 1, Proc: 1) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.960727] T [client-handshake.c:1664:select_server_supported_programs] 0-gv14-client-0: GF-DUMP (1) not supported
>[2013-03-07 17:00:46.960749] I [client-handshake.c:1658:select_server_supported_programs] 0-gv14-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-03-07 17:00:46.960803] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.960831] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 536, payload: 472, rpc hdr: 64
>[2013-03-07 17:00:46.960912] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_setvolume+0x460) [0x7fe879be03b0] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv14-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.960957] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.960987] D [client-handshake.c:185:client_start_ping] 0-gv14-client-0: returning as transport is already disconnected OR there are no frames (1 || 1)
>[2013-03-07 17:00:46.961038] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.961088] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.961200] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-1 is remote
>[2013-03-07 17:00:46.961235] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.961249] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.961289] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client3_3_getxattr+0x3bb) [0x7fe879bd349b] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv13-client-0: connect () called on transport already connected
>[2013-03-07 17:00:46.961316] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.961329] D [client-handshake.c:185:client_start_ping] 0-gv13-client-0: returning as transport is already disconnected OR there are no frames (2 || 1)
>[2013-03-07 17:00:46.962779] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.962853] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv14-client-0: clnt-lk-version = 1, server-lk-version = 0
>[2013-03-07 17:00:46.962882] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv14-client-0: Connected to 172.22.6.137:49153, attached to remote volume '/gfs0/brick14'.
>[2013-03-07 17:00:46.962906] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv14-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-03-07 17:00:46.962929] D [client-handshake.c:1318:client_post_handshake] 0-gv14-client-0: No fds to open - notifying all parents child up
>[2013-03-07 17:00:46.962959] D [client-handshake.c:486:client_set_lk_version] 0-gv14-client-0: Sending SET_LK_VERSION
>[2013-03-07 17:00:46.962990] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.963013] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64
>[2013-03-07 17:00:46.963049] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.963074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.963091] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-03-07 17:00:46.963146] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.963170] I [afr-common.c:3724:afr_notify] 0-gv14-replicate-0: Subvolume 'gv14-client-0' came back up; going online.
>[2013-03-07 17:00:46.963251] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.963289] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv13-client-0: clnt-lk-version = 1, server-lk-version = 0
>[2013-03-07 17:00:46.963304] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv13-client-0: Connected to 172.22.6.136:49152, attached to remote volume '/gfs0/brick13'.
>[2013-03-07 17:00:46.963318] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv13-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-03-07 17:00:46.963330] D [client-handshake.c:1318:client_post_handshake] 0-gv13-client-0: No fds to open - notifying all parents child up
>[2013-03-07 17:00:46.963326] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.963348] D [client-handshake.c:486:client_set_lk_version] 0-gv13-client-0: Sending SET_LK_VERSION
>[2013-03-07 17:00:46.963387] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.963403] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.963446] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64
>[2013-03-07 17:00:46.963448] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.963494] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.963556] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.963574] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-03-07 17:00:46.963609] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.963688] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.963696] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv14-client-0: Server lk version = 1
>[2013-03-07 17:00:46.963696] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.963705] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.963712] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.963723] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.963734] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv14-client-1: clnt-lk-version = 1, server-lk-version = 0
>[2013-03-07 17:00:46.963733] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.963738] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv14-client-1: Connected to 172.22.6.136:49153, attached to remote volume '/gfs0/brick14'.
>[2013-03-07 17:00:46.963737] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-0 is local
>[2013-03-07 17:00:46.963755] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv14-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-03-07 17:00:46.963808] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv13-replicate-0: Setting lk-owner=140636503968112
>[2013-03-07 17:00:46.963831] D [client-handshake.c:1318:client_post_handshake] 0-gv14-client-1: No fds to open - notifying all parents child up
>[2013-03-07 17:00:46.963855] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv13-replicate-0: starting crawl 1 for gv13-client-0
>[2013-03-07 17:00:46.963886] D [client-handshake.c:486:client_set_lk_version] 0-gv14-client-1: Sending SET_LK_VERSION
>[2013-03-07 17:00:46.963911] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.963929] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64
>[2013-03-07 17:00:46.963974] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.963989] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964000] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964021] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.964032] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-03-07 17:00:46.964064] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.964064] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.964071] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.964097] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.964124] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964131] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv13-client-0: Server lk version = 1
>[2013-03-07 17:00:46.964082] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.964135] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964160] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964143] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.964169] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.964180] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv14-client-1: Server lk version = 1
>[2013-03-07 17:00:46.964207] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.964240] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.964251] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-0 is remote
>[2013-03-07 17:00:46.964257] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964339] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964345] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.964349] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.964358] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.964364] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.964369] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv3-client-0: clnt-lk-version = 1, server-lk-version = 0
>[2013-03-07 17:00:46.964374] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv3-client-0: Connected to 172.22.6.136:49154, attached to remote volume '/gfs0/brick1'.
>[2013-03-07 17:00:46.964379] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv3-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-03-07 17:00:46.964384] D [client-handshake.c:1318:client_post_handshake] 0-gv3-client-0: No fds to open - notifying all parents child up
>[2013-03-07 17:00:46.964390] D [client-handshake.c:486:client_set_lk_version] 0-gv3-client-0: Sending SET_LK_VERSION
>[2013-03-07 17:00:46.964445] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964473] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64
>[2013-03-07 17:00:46.964489] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.964498] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964504] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-03-07 17:00:46.964514] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.964521] I [afr-common.c:3724:afr_notify] 0-gv3-replicate-0: Subvolume 'gv3-client-0' came back up; going online.
>[2013-03-07 17:00:46.964548] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964560] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964563] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-0 is local
>[2013-03-07 17:00:46.964566] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.964584] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv13-replicate-0: Setting lk-owner=140636503957692
>[2013-03-07 17:00:46.964576] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.964597] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964595] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv13-replicate-0: starting crawl 1 for gv13-client-0
>[2013-03-07 17:00:46.964615] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.964631] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.964633] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-1 is remote
>[2013-03-07 17:00:46.964645] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv3-client-0: Server lk version = 1
>[2013-03-07 17:00:46.964644] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964665] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.964666] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.964683] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.964689] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.964694] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964703] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.964709] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv13-replicate-0: Another crawl is in progress for gv13-client-0
>[2013-03-07 17:00:46.964723] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964744] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-1 is local
>[2013-03-07 17:00:46.964757] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv14-replicate-0: Setting lk-owner=140636503959776
>[2013-03-07 17:00:46.964763] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv14-replicate-0: starting crawl 1 for gv14-client-1
>[2013-03-07 17:00:46.964760] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.964789] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-0 is remote
>[2013-03-07 17:00:46.964790] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964811] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964823] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.964857] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.964862] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.964871] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-0 is local
>[2013-03-07 17:00:46.964874] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.964880] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv13-replicate-0: Setting lk-owner=140636503972280
>[2013-03-07 17:00:46.964891] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.964894] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv13-replicate-0: starting crawl 1 for gv13-client-0
>[2013-03-07 17:00:46.964938] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.964945] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.964952] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.964981] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.964983] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965002] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local
>[2013-03-07 17:00:46.965013] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503978532
>[2013-03-07 17:00:46.965012] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.965023] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0
>[2013-03-07 17:00:46.965032] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.965050] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.965073] D [afr-self-heald.c:1000:_crawl_directory] 0-gv13-replicate-0: crawling INDEX 51a9e1e3-32e0-4bd4-94ae-7ea06c6cf9ae
>[2013-03-07 17:00:46.965074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.965102] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.965107] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.965116] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.965122] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.965136] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.965138] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv13-replicate-0: Another crawl is in progress for gv13-client-0
>[2013-03-07 17:00:46.965173] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.965184] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.965183] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965197] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.965209] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-1 is local
>[2013-03-07 17:00:46.965222] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965224] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv14-replicate-0: Setting lk-owner=140636503970196
>[2013-03-07 17:00:46.965239] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv14-replicate-0: starting crawl 1 for gv14-client-1
>[2013-03-07 17:00:46.965251] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.965263] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.965293] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965365] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.965379] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv14-replicate-0: Another crawl is in progress for gv14-client-1
>[2013-03-07 17:00:46.965393] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-0: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-0)
>[2013-03-07 17:00:46.965406] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-0 is remote
>[2013-03-07 17:00:46.965407] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local
>[2013-03-07 17:00:46.965437] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.965438] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.965466] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.965466] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.965479] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.965448] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503972280
>[2013-03-07 17:00:46.965482] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965503] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0
>[2013-03-07 17:00:46.965509] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.965517] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.965538] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.965555] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.965563] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.965567] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-1: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-1)
>[2013-03-07 17:00:46.965570] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.965602] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.965609] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965573] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.965622] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.965663] T [socket.c:2643:socket_connect] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client3_3_getxattr+0x3bb) [0x7fe879bd349b] (-->/usr/lib64/glusterfs/3.4.0alpha/xlator/protocol/client.so(client_submit_request+0x153) [0x7fe879bc30c3] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x4b9) [0x3ddde0d1c9]))) 0-gv3-client-1: connect () called on transport already connected
>[2013-03-07 17:00:46.965665] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.965685] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.965687] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-0
>[2013-03-07 17:00:46.965698] D [client-handshake.c:185:client_start_ping] 0-gv3-client-1: returning as transport is already disconnected OR there are no frames (2 || 1)
>[2013-03-07 17:00:46.965717] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv13-replicate-0: child gv13-client-1 is remote
>[2013-03-07 17:00:46.965726] D [afr-self-heald.c:1000:_crawl_directory] 0-gv14-replicate-0: crawling INDEX c34b72c4-a2ff-4056-bea4-6d8eb09a45d9
>[2013-03-07 17:00:46.965718] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965743] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.965753] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.965757] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv14-replicate-0: child gv14-client-1 is local
>[2013-03-07 17:00:46.965763] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.965771] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv14-replicate-0: Setting lk-owner=140636503974364
>[2013-03-07 17:00:46.965773] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.965784] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv14-replicate-0: starting crawl 1 for gv14-client-1
>[2013-03-07 17:00:46.965781] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv3-client-1: clnt-lk-version = 1, server-lk-version = 0
>[2013-03-07 17:00:46.965776] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv13-replicate-0: Crawl completed on gv13-client-0
>[2013-03-07 17:00:46.965797] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv3-client-1: Connected to 172.22.6.136:49155, attached to remote volume '/gfs0/brick2'.
>[2013-03-07 17:00:46.965841] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv3-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-03-07 17:00:46.965852] D [client-handshake.c:1318:client_post_handshake] 0-gv3-client-1: No fds to open - notifying all parents child up
>[2013-03-07 17:00:46.965873] D [client-handshake.c:486:client_set_lk_version] 0-gv3-client-1: Sending SET_LK_VERSION
>[2013-03-07 17:00:46.965877] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.965888] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.965909] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.965918] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64
>[2013-03-07 17:00:46.965980] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.965987] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.966005] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.966035] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-03-07 17:00:46.966048] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.966195] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.966228] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.966244] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.966259] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.966272] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x5x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.966315] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.966314] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.966399] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.966403] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.966446] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.966453] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x4x Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.966462] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local
>[2013-03-07 17:00:46.966465] D [client-handshake.c:1430:client_setvolume_cbk] 0-gv3-client-2: clnt-lk-version = 1, server-lk-version = 0
>[2013-03-07 17:00:46.966469] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503963944
>[2013-03-07 17:00:46.966471] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv3-client-2: Connected to 172.22.6.136:49156, attached to remote volume '/gfs0/brick3'.
>[2013-03-07 17:00:46.966477] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1
>[2013-03-07 17:00:46.966473] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.966482] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv3-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-03-07 17:00:46.966513] D [client-handshake.c:1318:client_post_handshake] 0-gv3-client-2: No fds to open - notifying all parents child up
>[2013-03-07 17:00:46.966524] D [client-handshake.c:486:client_set_lk_version] 0-gv3-client-2: Sending SET_LK_VERSION
>[2013-03-07 17:00:46.966533] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.966539] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 156, payload: 92, rpc hdr: 64
>[2013-03-07 17:00:46.966556] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.966530] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac
>[2013-03-07 17:00:46.966570] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.966598] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 64, payload: 0, rpc hdr: 64
>[2013-03-07 17:00:46.966601] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.966616] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.966623] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.966677] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.966744] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.966778] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.966773] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.966800] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.966816] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv3-client-1: Server lk version = 1
>[2013-03-07 17:00:46.966814] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.966826] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x5x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.966867] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv3-client-2: Server lk version = 1
>[2013-03-07 17:00:46.966882] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.966916] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x7x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.966939] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x6x Program: GlusterFS Handshake, ProgVers: 2, Proc: 3) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.966940] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.966956] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.966961] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.966983] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.967005] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.967020] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.967026] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.967052] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv14-replicate-0: Crawl completed on gv14-client-1
>[2013-03-07 17:00:46.967097] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.967115] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.967154] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.967154] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.967215] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.967231] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x7x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.967236] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.967285] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.967317] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local
>[2013-03-07 17:00:46.967345] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503986868
>[2013-03-07 17:00:46.967360] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1
>[2013-03-07 17:00:46.967373] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.967491] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.967833] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.967865] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.967911] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.967939] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0
>[2013-03-07 17:00:46.967976] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.967994] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.968032] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.968070] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv14-replicate-0: Another crawl is in progress for gv14-client-1
>[2013-03-07 17:00:46.968130] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local
>[2013-03-07 17:00:46.968148] D [afr-self-heald.c:1000:_crawl_directory] 0-gv13-replicate-0: crawling INDEX 51a9e1e3-32e0-4bd4-94ae-7ea06c6cf9ae
>[2013-03-07 17:00:46.968192] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.968196] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503970196
>[2013-03-07 17:00:46.968190] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.968226] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.968349] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.968254] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.968293] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2
>[2013-03-07 17:00:46.968386] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.968247] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.968351] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.968479] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.968526] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.968527] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.968560] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.968573] D [afr-self-heald.c:1000:_crawl_directory] 0-gv14-replicate-0: crawling INDEX c34b72c4-a2ff-4056-bea4-6d8eb09a45d9
>[2013-03-07 17:00:46.968585] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.968627] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.968633] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.968664] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.968678] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.968657] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.968678] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.968759] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.968683] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-1
>[2013-03-07 17:00:46.968656] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.968726] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.968728] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.968821] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.969122] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.969153] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local
>[2013-03-07 17:00:46.969130] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.969184] T [syncop.c:608:syncop_readdir_cbk] 0-gv13-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.968826] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.969180] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503963944
>[2013-03-07 17:00:46.969213] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.969217] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0
>[2013-03-07 17:00:46.969247] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.969230] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv13-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.969264] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.969282] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.969279] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.969288] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.969317] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac
>[2013-03-07 17:00:46.969320] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.969332] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.969340] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.969343] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.969349] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.969349] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.969396] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.969395] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.969394] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.969433] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x8x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.969462] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.969476] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-0
>[2013-03-07 17:00:46.969501] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.969472] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.969549] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv13-client-0: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv13-client-0)
>[2013-03-07 17:00:46.969522] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.969599] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.969603] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv13-replicate-0: Crawl completed on gv13-client-0
>[2013-03-07 17:00:46.969532] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.969494] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.969609] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.969712] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.969623] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.969540] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-0 is local
>[2013-03-07 17:00:46.969748] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.969765] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.969752] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.969811] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.969772] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503976448
>[2013-03-07 17:00:46.969816] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.969828] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-0
>[2013-03-07 17:00:46.969837] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.969853] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.969860] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x9x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.969864] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv14-replicate-0: Crawl completed on gv14-client-1
>[2013-03-07 17:00:46.969921] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.969894] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local
>[2013-03-07 17:00:46.969976] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.970002] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX cfb7526b-cb8e-4c9a-90c1-0009a48b9d27
>[2013-03-07 17:00:46.970031] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.969983] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503982700
>[2013-03-07 17:00:46.970022] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.970065] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.970071] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2
>[2013-03-07 17:00:46.970124] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.970085] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0
>[2013-03-07 17:00:46.970151] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.970080] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x10x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.970190] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.970221] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.970242] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.970252] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX 484a792f-dd0b-4c99-9d77-3cb20c96c621
>[2013-03-07 17:00:46.970294] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local
>[2013-03-07 17:00:46.970260] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.970299] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.970205] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-2
>[2013-03-07 17:00:46.970393] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.970301] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.970325] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503953524
>[2013-03-07 17:00:46.970374] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.970485] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.970494] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.970531] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.970525] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.970483] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1
>[2013-03-07 17:00:46.970535] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.970617] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.970645] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=78a027b5-dae3-4851-904c-926034438fb4, count=2
>[2013-03-07 17:00:46.970645] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.970695] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.970702] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-1
>[2013-03-07 17:00:46.970702] D [afr-self-heald.c:412:_self_heal_entry] 0-gv3-replicate-0: lookup <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.970722] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.970722] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.970717] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.970814] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.970830] D [afr-self-heald.c:1000:_crawl_directory] 0-gv14-replicate-0: crawling INDEX c34b72c4-a2ff-4056-bea4-6d8eb09a45d9
>[2013-03-07 17:00:46.970835] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.970872] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.970835] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-gv3-replicate-0: <gfid:78a027b5-dae3-4851-904c-926034438fb4>: failed to get the gfid from dict
>[2013-03-07 17:00:46.970892] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.970929] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.970962] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.970971] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.970981] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.971026] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.971023] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68
>[2013-03-07 17:00:46.971036] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.971037] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.971099] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=78a027b5-dae3-4851-904c-926034438fb4, count=2
>[2013-03-07 17:00:46.971112] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971150] D [afr-self-heald.c:412:_self_heal_entry] 0-gv3-replicate-0: lookup <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.971157] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971180] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.971199] D [afr-common.c:131:afr_lookup_xattr_req_prepare] 0-gv3-replicate-0: <gfid:78a027b5-dae3-4851-904c-926034438fb4>: failed to get the gfid from dict
>[2013-03-07 17:00:46.971200] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.971206] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68
>[2013-03-07 17:00:46.971217] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.971218] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.971216] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac
>[2013-03-07 17:00:46.971262] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.971251] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68
>[2013-03-07 17:00:46.971299] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.971241] T [syncop.c:608:syncop_readdir_cbk] 0-gv14-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.971321] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.971325] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971356] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x12x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.971325] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.971378] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.971359] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68
>[2013-03-07 17:00:46.971388] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68
>[2013-03-07 17:00:46.971408] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971384] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv14-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: e068dc7ce87f0000
>[2013-03-07 17:00:46.971429] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.971443] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.971461] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971466] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local
>[2013-03-07 17:00:46.971488] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.971503] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.971576] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971553] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503963944
>[2013-03-07 17:00:46.971584] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 356, payload: 288, rpc hdr: 68
>[2013-03-07 17:00:46.971598] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-0
>[2013-03-07 17:00:46.971640] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.971611] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2
>[2013-03-07 17:00:46.971585] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.971613] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.971629] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971781] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.971787] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.971810] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv14-client-1: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv14-client-1)
>[2013-03-07 17:00:46.971814] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.971824] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.971840] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.971833] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv14-replicate-0: Crawl completed on gv14-client-1
>[2013-03-07 17:00:46.971954] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.972040] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.972089] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.972093] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x13x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.972086] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0
>[2013-03-07 17:00:46.972165] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.972217] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.972218] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.972246] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.972277] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.972314] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.972353] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.972368] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.972373] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 1
>[2013-03-07 17:00:46.972443] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 1 for <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.972455] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 1
>[2013-03-07 17:00:46.972463] D [afr-common.c:1282:afr_detect_self_heal_by_iatt] 0-gv3-replicate-0: size differs for <gfid:78a027b5-dae3-4851-904c-926034438fb4> 
>[2013-03-07 17:00:46.972469] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972474] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972479] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972486] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972493] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.972499] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.972504] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.972509] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.972515] D [afr-common.c:1443:afr_launch_self_heal] 0-gv3-replicate-0: background  meta-data data self-heal triggered. path: <gfid:78a027b5-dae3-4851-904c-926034438fb4>, reason: lookup detected pending operations
>[2013-03-07 17:00:46.972524] T [afr-self-heal-common.c:2267:afr_self_heal] 0-gv3-replicate-0: performing self heal on <gfid:78a027b5-dae3-4851-904c-926034438fb4> (metadata=1 data=1 entry=0)
>[2013-03-07 17:00:46.972545] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503968112
>[2013-03-07 17:00:46.972580] T [afr-self-heal-common.c:2357:afr_self_heal] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.972609] T [afr-self-heal-common.c:1027:afr_sh_missing_entries_done] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.972661] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.972681] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.972734] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.972769] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.972786] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.972823] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.972835] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.972841] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.972854] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.972873] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.972893] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x14x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.972907] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.972913] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.972920] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.972925] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.972930] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 2
>[2013-03-07 17:00:46.972935] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 2 for <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.972941] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 2
>[2013-03-07 17:00:46.972947] D [afr-common.c:1282:afr_detect_self_heal_by_iatt] 0-gv3-replicate-0: size differs for <gfid:78a027b5-dae3-4851-904c-926034438fb4> 
>[2013-03-07 17:00:46.972953] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972958] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972963] D [afr-common.c:1211:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: metadata self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972969] D [afr-common.c:1225:afr_lookup_set_self_heal_params_by_xattr] 0-gv3-replicate-0: data self-heal is pending for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.972975] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.972993] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.973002] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.973007] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.973012] D [afr-common.c:1443:afr_launch_self_heal] 0-gv3-replicate-0: background  meta-data data self-heal triggered. path: <gfid:78a027b5-dae3-4851-904c-926034438fb4>, reason: lookup detected pending operations
>[2013-03-07 17:00:46.973018] T [afr-self-heal-common.c:2267:afr_self_heal] 0-gv3-replicate-0: performing self heal on <gfid:78a027b5-dae3-4851-904c-926034438fb4> (metadata=1 data=1 entry=0)
>[2013-03-07 17:00:46.973024] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503957692
>[2013-03-07 17:00:46.973038] T [afr-self-heal-common.c:2357:afr_self_heal] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.973070] T [afr-self-heal-common.c:1027:afr_sh_missing_entries_done] 0-gv3-replicate-0: proceeding to metadata check on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.973097] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.973114] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.973157] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.973189] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.973206] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.973242] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.973253] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.973259] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.973273] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.973288] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.973310] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.973318] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.973324] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x15x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.973331] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.973348] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.973360] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.973356] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-1 is local
>[2013-03-07 17:00:46.973397] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503959776
>[2013-03-07 17:00:46.973429] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-1
>[2013-03-07 17:00:46.973377] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.973525] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received
>[2013-03-07 17:00:46.973524] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.973524] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-1
>[2013-03-07 17:00:46.973581] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 120, payload: 56, rpc hdr: 64
>[2013-03-07 17:00:46.973559] T [afr-lk-common.c:1477:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: All servers locked. Calling the cbk
>[2013-03-07 17:00:46.973626] D [afr-self-heal-metadata.c:486:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking metadata inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. Proceeding to FOP
>[2013-03-07 17:00:46.973662] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.973668] D [afr-self-heal-common.c:1930:afr_sh_common_lookup] 0-gv3-replicate-0: looking up <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-0
>[2013-03-07 17:00:46.973753] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.973761] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 252, payload: 184, rpc hdr: 68
>[2013-03-07 17:00:46.973784] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.973794] D [afr-self-heal-common.c:1930:afr_sh_common_lookup] 0-gv3-replicate-0: looking up <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-1
>[2013-03-07 17:00:46.973804] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.973810] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 252, payload: 184, rpc hdr: 68
>[2013-03-07 17:00:46.973828] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.973836] D [afr-self-heal-common.c:1930:afr_sh_common_lookup] 0-gv3-replicate-0: looking up <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-2
>[2013-03-07 17:00:46.973847] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.973853] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 252, payload: 184, rpc hdr: 68
>[2013-03-07 17:00:46.973867] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.973886] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.973898] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.973910] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x16x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.973924] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received
>[2013-03-07 17:00:46.973930] T [afr-lk-common.c:1486:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: 0 servers locked. Trying again with blocking calls
>[2013-03-07 17:00:46.973937] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.973944] T [afr-lk-common.c:716:afr_unlock_inodelk] 0-gv3-replicate-0: No internal locks unlocked
>[2013-03-07 17:00:46.973950] D [afr-self-heal-metadata.c:478:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking metadata inodelks failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.973956] D [afr-self-heal-metadata.c:480:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Metadata self-heal failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>.
>[2013-03-07 17:00:46.973961] D [afr-self-heal-metadata.c:61:afr_sh_metadata_done] 0-gv3-replicate-0: proceeding to data check on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.973979] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.973986] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-03-07 17:00:46.974001] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.974044] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.974060] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-03-07 17:00:46.974094] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.974106] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.974112] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-03-07 17:00:46.974126] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.974141] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.974160] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x17x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.974164] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX c1d6ffd3-397a-45d2-b72c-f7d177a204ac
>[2013-03-07 17:00:46.974179] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.974178] D [afr-self-heald.c:1084:afr_find_child_position] 0-gv3-replicate-0: child gv3-client-2 is local
>[2013-03-07 17:00:46.974188] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.974183] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.974194] T [afr-lk-common.c:113:afr_set_lk_owner] 0-gv3-replicate-0: Setting lk-owner=140636503953524
>[2013-03-07 17:00:46.974205] D [afr-self-heald.c:1233:afr_start_crawl] 0-gv3-replicate-0: starting crawl 1 for gv3-client-2
>[2013-03-07 17:00:46.974205] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.974212] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.974261] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.974286] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-0
>[2013-03-07 17:00:46.974434] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x18x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.974475] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.974496] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.974510] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.974525] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.974540] T [afr-self-heal-metadata.c:355:afr_sh_metadata_sync_prepare] 0-gv3-replicate-0: syncing metadata of <gfid:78a027b5-dae3-4851-904c-926034438fb4> from subvolume gv3-client-1 to 1 active sinks
>[2013-03-07 17:00:46.974574] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.974596] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.974640] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-2
>[2013-03-07 17:00:46.974659] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.974712] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.974738] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-1
>[2013-03-07 17:00:46.974789] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.974817] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.974832] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.974886] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x19x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.974911] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-2
>[2013-03-07 17:00:46.974907] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 24b2dc7ce87f0000
>[2013-03-07 17:00:46.974927] T [afr-self-heal-data.c:1280:afr_sh_data_open_cbk] 0-gv3-replicate-0: fd for <gfid:78a027b5-dae3-4851-904c-926034438fb4> opened, commencing sync
>[2013-03-07 17:00:46.974955] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.974987] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.975009] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.975015] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.975105] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.975160] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.975178] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.975220] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.975250] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.975265] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.975301] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.975352] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.975364] D [afr-self-heal-metadata.c:267:afr_sh_metadata_sync] 0-gv3-replicate-0: self-healing metadata of <gfid:78a027b5-dae3-4851-904c-926034438fb4> from gv3-client-1 to gv3-client-0
>[2013-03-07 17:00:46.975376] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.975382] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68
>[2013-03-07 17:00:46.975398] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.975418] T [client.c:1176:is_client_rpc_init_command] 0-gv3-client-0: name is !replace-brick
>[2013-03-07 17:00:46.975425] T [client.c:1201:is_client_rpc_destroy_command] 0-gv3-client-0: name is !replace-brick
>[2013-03-07 17:00:46.975434] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.975440] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 100, payload: 32, rpc hdr: 68
>[2013-03-07 17:00:46.975453] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 17) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.975496] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.975542] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.975541] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-0
>[2013-03-07 17:00:46.975579] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.975726] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x20x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.975772] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received
>[2013-03-07 17:00:46.975789] T [afr-lk-common.c:1486:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: 0 servers locked. Trying again with blocking calls
>[2013-03-07 17:00:46.975817] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.975832] T [afr-lk-common.c:716:afr_unlock_inodelk] 0-gv3-replicate-0: No internal locks unlocked
>[2013-03-07 17:00:46.975849] D [afr-self-heal-data.c:1134:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. by bc60dc7ce87f0000
>[2013-03-07 17:00:46.975885] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.975904] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.975961] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.976006] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.976049] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 17) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.976090] D [afr-self-heal-common.c:2510:afr_sh_erase_pending] 0-gv3-replicate-0: Delta matrix for: 7089dc7ce87f0000
>[2013-03-07 17:00:46.976107] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.976121] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ]
>[2013-03-07 17:00:46.976134] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ]
>[2013-03-07 17:00:46.976164] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.976183] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 236, payload: 168, rpc hdr: 68
>[2013-03-07 17:00:46.976223] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.976258] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.976275] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 236, payload: 168, rpc hdr: 68
>[2013-03-07 17:00:46.976315] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.976347] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.976363] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 236, payload: 168, rpc hdr: 68
>[2013-03-07 17:00:46.976400] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.976453] I [afr-self-heald.c:1180:afr_dir_exclusive_crawl] 0-gv3-replicate-0: Another crawl is in progress for gv3-client-2
>[2013-03-07 17:00:46.976591] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.976903] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.976975] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x21x Program: GlusterFS 3.3, ProgVers: 330, Proc: 33) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.977015] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.977046] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.977065] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.977123] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.977158] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.977175] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.977217] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.977247] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.977264] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.977303] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.977405] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.977447] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.977472] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.977524] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.977580] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.977618] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.977648] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x22x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.977665] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked
>[2013-03-07 17:00:46.977685] D [afr-self-heal-metadata.c:61:afr_sh_metadata_done] 0-gv3-replicate-0: proceeding to data check on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.977717] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.977738] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-03-07 17:00:46.977778] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.977809] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.977826] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-03-07 17:00:46.977864] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.977894] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.977923] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 24, rpc hdr: 68
>[2013-03-07 17:00:46.977960] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.978010] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.978021] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.978027] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.978039] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.978051] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.978059] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-0
>[2013-03-07 17:00:46.978147] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.978189] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-1
>[2013-03-07 17:00:46.978219] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x23x Program: GlusterFS 3.3, ProgVers: 330, Proc: 11) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.978239] T [afr-self-heal-data.c:1265:afr_sh_data_open_cbk] 0-gv3-replicate-0: open of <gfid:78a027b5-dae3-4851-904c-926034438fb4> succeeded on child gv3-client-2
>[2013-03-07 17:00:46.978253] T [afr-self-heal-data.c:1280:afr_sh_data_open_cbk] 0-gv3-replicate-0: fd for <gfid:78a027b5-dae3-4851-904c-926034438fb4> opened, commencing sync
>[2013-03-07 17:00:46.978278] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.978294] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.978335] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.978368] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.978384] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.978433] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.978465] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.978481] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.978519] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.978559] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x24x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.978581] D [afr-lk-common.c:1098:afr_lock_blocking] 0-gv3-replicate-0: we're done locking
>[2013-03-07 17:00:46.978593] D [afr-self-heal-data.c:1112:afr_sh_data_post_blocking_inodelk_cbk] 0-gv3-replicate-0: Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by bc60dc7ce87f0000. Proceding to self-heal
>[2013-03-07 17:00:46.978610] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.978617] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.978643] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.978657] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.978664] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.978679] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.978716] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.978733] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.978767] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.978812] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x25x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.978845] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.978874] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.978891] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received
>[2013-03-07 17:00:46.978905] T [afr-lk-common.c:1486:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: 0 servers locked. Trying again with blocking calls
>[2013-03-07 17:00:46.978919] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.978931] T [afr-lk-common.c:716:afr_unlock_inodelk] 0-gv3-replicate-0: No internal locks unlocked
>[2013-03-07 17:00:46.978947] D [afr-self-heal-data.c:1134:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks failed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>. by 7089dc7ce87f0000
>[2013-03-07 17:00:46.978971] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.978988] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.979028] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.979079] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.979108] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded
>[2013-03-07 17:00:46.979139] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.979171] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded
>[2013-03-07 17:00:46.979216] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x26x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.979242] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded
>[2013-03-07 17:00:46.979273] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.979291] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.979327] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.979356] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.979373] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.979405] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.979462] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.979480] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.979514] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.979691] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.979726] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded
>[2013-03-07 17:00:46.979756] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.979775] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded
>[2013-03-07 17:00:46.979905] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x27x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.979934] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded
>[2013-03-07 17:00:46.979952] D [afr-self-heal-data.c:692:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: Pending matrix for: bc60dc7ce87f0000
>[2013-03-07 17:00:46.979974] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.979989] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.980002] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.980015] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.980033] D [afr-self-heal-data.c:672:afr_sh_data_fix] 0-gv3-replicate-0: self-healing file <gfid:78a027b5-dae3-4851-904c-926034438fb4> from subvolume gv3-client-1 to 1 other
>[2013-03-07 17:00:46.980071] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.980089] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 104, payload: 36, rpc hdr: 68
>[2013-03-07 17:00:46.980137] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 24) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.980443] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 24) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.980476] D [afr-self-heal-data.c:547:afr_sh_data_trim_cbk] 0-gv3-replicate-0: ftruncate of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on subvolume gv3-client-0 completed
>[2013-03-07 17:00:46.980519] T [afr-self-heal-algorithm.c:350:sh_loop_driver] 0-gv3-replicate-0: spawning a loop for offset 0
>[2013-03-07 17:00:46.980558] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.980577] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.980620] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.980652] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.980667] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.980706] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.980735] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.980750] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.980787] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.980944] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.981158] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.981191] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x28x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.981208] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received
>[2013-03-07 17:00:46.981222] T [afr-lk-common.c:1477:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: All servers locked. Calling the cbk
>[2013-03-07 17:00:46.981238] D [afr-self-heal-data.c:1147:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by bc60dc7ce87f0000. Proceeding to self-heal
>[2013-03-07 17:00:46.981257] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.981280] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.981296] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.981337] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.981378] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.981395] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.981445] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.981476] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.981492] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.981528] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.981549] D [afr-self-heal-algorithm.c:168:sh_loop_lock_success] 0-gv3-replicate-0: Acquired lock for range 0 131072
>[2013-03-07 17:00:46.981579] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.981597] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.981628] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 12) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.981664] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.981924] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x29x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.981956] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.981973] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked
>[2013-03-07 17:00:46.982763] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 12) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.982793] T [afr-self-heal-algorithm.c:513:sh_loop_read_cbk] 0-gv3-replicate-0: read 1979 bytes of data from <gfid:78a027b5-dae3-4851-904c-926034438fb4>, offset 0
>[2013-03-07 17:00:46.982822] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.982838] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.982884] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.983207] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.983237] T [afr-self-heal-algorithm.c:426:sh_loop_write_cbk] 0-gv3-replicate-0: wrote 1979 bytes of data from <gfid:78a027b5-dae3-4851-904c-926034438fb4> to child 0, offset 0
>[2013-03-07 17:00:46.983256] T [afr-self-heal-algorithm.c:383:sh_loop_return] 0-gv3-replicate-0: loop for offset 0 returned
>[2013-03-07 17:00:46.983269] D [afr-self-heal-algorithm.c:115:sh_loop_driver_done] 0-gv3-replicate-0: full self-heal completed on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.983303] D [afr-self-heal-common.c:2510:afr_sh_erase_pending] 0-gv3-replicate-0: Delta matrix for: bc60dc7ce87f0000
>[2013-03-07 17:00:46.983319] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.983347] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ]
>[2013-03-07 17:00:46.983363] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ -2 0 0 ]
>[2013-03-07 17:00:46.983390] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.983418] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.983462] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.983496] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.983513] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.983549] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.983580] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.983596] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.983632] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.983723] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.984020] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.984064] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x30x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.984098] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.984114] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.984153] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.984184] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.984200] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.984236] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.984267] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.984282] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.984320] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.984473] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.984696] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.984741] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x31x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.984758] T [afr-lk-common.c:1472:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Last inode locking reply received
>[2013-03-07 17:00:46.984771] T [afr-lk-common.c:1477:afr_nonblocking_inodelk_cbk] 0-gv3-replicate-0: All servers locked. Calling the cbk
>[2013-03-07 17:00:46.984787] D [afr-self-heal-data.c:1147:afr_sh_data_post_nonblocking_inodelk_cbk] 0-gv3-replicate-0: Non Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by bc60dc7ce87f0000. Proceeding to self-heal
>[2013-03-07 17:00:46.984802] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.984825] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.984841] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.984880] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.984909] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.984925] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.984962] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.984990] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.985006] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.985039] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.985089] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.985108] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.985145] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.985177] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.985194] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.985232] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.985263] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.985283] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.985315] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.985359] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.985433] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.985493] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x32x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.985515] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked
>[2013-03-07 17:00:46.985565] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.985592] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded
>[2013-03-07 17:00:46.985655] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.985685] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded
>[2013-03-07 17:00:46.985759] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x33x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.985787] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded
>[2013-03-07 17:00:46.985813] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.985828] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.985865] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.985894] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.985909] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.985943] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.985970] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.985985] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.986018] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.986084] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x52x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.986109] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded
>[2013-03-07 17:00:46.986351] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x34x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.986374] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded
>[2013-03-07 17:00:46.986399] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.986443] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded
>[2013-03-07 17:00:46.986483] D [afr-self-heal-data.c:692:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: Pending matrix for: bc60dc7ce87f0000
>[2013-03-07 17:00:46.986504] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.986518] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.986531] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.986545] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 0
>[2013-03-07 17:00:46.986573] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.986590] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68
>[2013-03-07 17:00:46.986631] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.986663] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.986679] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68
>[2013-03-07 17:00:46.986714] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.986742] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.986758] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 192, payload: 124, rpc hdr: 68
>[2013-03-07 17:00:46.986795] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.986943] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x53x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.987213] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.987250] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x35x Program: GlusterFS 3.3, ProgVers: 330, Proc: 38) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.987271] D [afr-self-heal-data.c:325:afr_sh_data_finish] 0-gv3-replicate-0: finishing data selfheal of <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.987286] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.987309] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.987325] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.987365] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.987394] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.987420] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.987459] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.987489] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.987514] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.987552] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.987665] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.987698] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.987713] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.987751] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.987796] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x54x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.987905] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x36x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.987950] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.987969] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked
>[2013-03-07 17:00:46.987984] D [afr-self-heal-data.c:143:afr_sh_data_close] 0-gv3-replicate-0: closing fd of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0
>[2013-03-07 17:00:46.988013] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.988029] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.988066] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.988087] D [afr-self-heal-data.c:143:afr_sh_data_close] 0-gv3-replicate-0: closing fd of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1
>[2013-03-07 17:00:46.988110] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.988125] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.988162] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.988182] D [afr-self-heal-data.c:143:afr_sh_data_close] 0-gv3-replicate-0: closing fd of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2
>[2013-03-07 17:00:46.988204] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: bc60dc7ce87f0000
>[2013-03-07 17:00:46.988219] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.988254] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.988294] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.988328] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.988344] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.988388] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.988434] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x55x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.988454] T [client-lk.c:391:delete_granted_locks_owner] 0-gv3-client-0: Number of locks cleared=0
>[2013-03-07 17:00:46.988473] T [client-rpc-fops.c:924:client3_3_flush_cbk] 0-gv3-client-0: deleting locks of owner (bc60dc7ce87f0000) returned 0
>[2013-03-07 17:00:46.988551] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x37x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.988572] T [client-lk.c:391:delete_granted_locks_owner] 0-gv3-client-2: Number of locks cleared=0
>[2013-03-07 17:00:46.988588] T [client-rpc-fops.c:924:client3_3_flush_cbk] 0-gv3-client-2: deleting locks of owner (bc60dc7ce87f0000) returned 0
>[2013-03-07 17:00:46.988616] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 15) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.988633] T [client-lk.c:391:delete_granted_locks_owner] 0-gv3-client-1: Number of locks cleared=0
>[2013-03-07 17:00:46.988649] T [client-rpc-fops.c:924:client3_3_flush_cbk] 0-gv3-client-1: deleting locks of owner (bc60dc7ce87f0000) returned 0
>[2013-03-07 17:00:46.988664] D [afr-self-heal-common.c:2216:afr_self_heal_completion_cbk] 0-gv3-replicate-0: background  meta-data data self-heal completed on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.988688] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.988703] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.988716] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.988730] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.988744] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 2
>[2013-03-07 17:00:46.988756] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 2 for <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.988773] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 2
>[2013-03-07 17:00:46.988834] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-0: sending release on fd
>[2013-03-07 17:00:46.988857] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.988874] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-03-07 17:00:46.988895] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.988914] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x56x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.988946] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 116, payload: 48, rpc hdr: 68
>[2013-03-07 17:00:46.988966] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-1: sending release on fd
>[2013-03-07 17:00:46.988999] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.989006] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-03-07 17:00:46.989023] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.989043] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-2: sending release on fd
>[2013-03-07 17:00:46.989016] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.989115] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-03-07 17:00:46.989149] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989192] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x38x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989213] D [afr-lk-common.c:1098:afr_lock_blocking] 0-gv3-replicate-0: we're done locking
>[2013-03-07 17:00:46.989230] D [afr-self-heal-data.c:1112:afr_sh_data_post_blocking_inodelk_cbk] 0-gv3-replicate-0: Blocking data inodelks done for <gfid:78a027b5-dae3-4851-904c-926034438fb4> by 7089dc7ce87f0000. Proceding to self-heal
>[2013-03-07 17:00:46.989274] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.989292] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.989330] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x57x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.989362] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.989378] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.989426] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.989460] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.989476] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 244, payload: 176, rpc hdr: 68
>[2013-03-07 17:00:46.989509] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989550] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x56x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.989581] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x39x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989627] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.989657] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x40x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989686] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x57x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.989694] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.989710] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded
>[2013-03-07 17:00:46.989732] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.989812] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989837] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.989867] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded
>[2013-03-07 17:00:46.989949] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x41x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.989974] T [afr-self-heal-data.c:939:afr_sh_common_fxattrop_resp_handler] 0-gv3-replicate-0: fxattrop of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded
>[2013-03-07 17:00:46.989999] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.990015] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.990054] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x58x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.990083] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.990099] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.990135] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.990163] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.990179] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
>[2013-03-07 17:00:46.990215] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.990253] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x42x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.990293] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x58x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.990318] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-0 succeeded
>[2013-03-07 17:00:46.990318] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-2
>[2013-03-07 17:00:46.990379] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.990381] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.990429] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-1 succeeded
>[2013-03-07 17:00:46.990434] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.990447] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x43x Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.990498] T [afr-self-heal-data.c:839:afr_sh_data_fstat_cbk] 0-gv3-replicate-0: fstat of <gfid:78a027b5-dae3-4851-904c-926034438fb4> on gv3-client-2 succeeded
>[2013-03-07 17:00:46.990511] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.990518] D [afr-self-heal-data.c:692:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: Pending matrix for: 7089dc7ce87f0000
>[2013-03-07 17:00:46.990588] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.990603] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.990616] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.990630] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 0
>[2013-03-07 17:00:46.990646] D [afr-self-heal-data.c:738:afr_sh_data_fxattrop_fstat_done] 0-gv3-replicate-0: No self-heal needed for <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.990660] D [afr-self-heal-data.c:325:afr_sh_data_finish] 0-gv3-replicate-0: finishing data selfheal of <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.990673] D [afr-lk-common.c:456:transaction_lk_op] 0-gv3-replicate-0: lk op is for a self heal
>[2013-03-07 17:00:46.990698] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.990715] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.990759] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x59x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.990789] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.990805] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.990840] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.990869] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 7089dc7ce87f0000
>[2013-03-07 17:00:46.990885] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 148, payload: 80, rpc hdr: 68
>[2013-03-07 17:00:46.990919] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.990957] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.991049] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x59x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.991074] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.991119] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.991143] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.991197] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x44x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.991293] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 29) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.991316] T [afr-lk-common.c:646:afr_unlock_common_cbk] 0-gv3-replicate-0: All internal locks unlocked
>[2013-03-07 17:00:46.991331] D [afr-self-heal-common.c:2216:afr_self_heal_completion_cbk] 0-gv3-replicate-0: background  meta-data data self-heal completed on <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.991354] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 0 0 0 ]
>[2013-03-07 17:00:46.991369] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.991382] D [afr-self-heal-common.c:138:afr_sh_print_pending_matrix] 0-gv3-replicate-0: pending_matrix: [ 2 0 0 ]
>[2013-03-07 17:00:46.991395] D [afr-self-heal-common.c:887:afr_mark_sources] 0-gv3-replicate-0: Number of sources: 2
>[2013-03-07 17:00:46.991416] D [afr-self-heal-data.c:814:afr_lookup_select_read_child_by_txn_type] 0-gv3-replicate-0: returning read_child: 1
>[2013-03-07 17:00:46.991431] D [afr-common.c:1397:afr_lookup_select_read_child] 0-gv3-replicate-0: Source selected as 1 for <gfid:78a027b5-dae3-4851-904c-926034438fb4>
>[2013-03-07 17:00:46.991447] D [afr-common.c:1134:afr_lookup_build_response_params] 0-gv3-replicate-0: Building lookup response from 1
>[2013-03-07 17:00:46.991499] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-0: sending release on fd
>[2013-03-07 17:00:46.991521] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-0: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.991550] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-03-07 17:00:46.991561] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.991568] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x60x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.991581] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 116, payload: 48, rpc hdr: 68
>[2013-03-07 17:00:46.991588] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-1: sending release on fd
>[2013-03-07 17:00:46.991597] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.991603] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-03-07 17:00:46.991608] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.991628] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.991655] D [client-rpc-fops.c:2789:client_fdctx_destroy] 0-gv3-client-2: sending release on fd
>[2013-03-07 17:00:46.991675] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 0, uid: 0, gid: 0, owner: 
>[2013-03-07 17:00:46.991690] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 92, payload: 28, rpc hdr: 64
>[2013-03-07 17:00:46.991728] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.991779] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.991824] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-0: received rpc message (RPC XID: 0x60x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-0)
>[2013-03-07 17:00:46.991839] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX 484a792f-dd0b-4c99-9d77-3cb20c96c621
>[2013-03-07 17:00:46.991905] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.991927] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.991993] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.992030] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.992067] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x45x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.992175] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x46x Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.992162] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.992216] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.992269] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.992374] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.992403] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.992428] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.992482] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-2: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: 9491dc7ce87f0000
>[2013-03-07 17:00:46.992530] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x47x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.992528] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.992551] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-1
>[2013-03-07 17:00:46.992593] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.992611] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.992654] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 124, payload: 56, rpc hdr: 68
>[2013-03-07 17:00:46.992690] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.992809] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-2: received rpc message (RPC XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-2)
>[2013-03-07 17:00:46.992894] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-2
>[2013-03-07 17:00:46.992958] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x48x Program: GlusterFS 3.3, ProgVers: 330, Proc: 18) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.993020] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.993059] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 112, payload: 44, rpc hdr: 68
>[2013-03-07 17:00:46.993109] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.993368] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x49x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.993433] D [afr-self-heald.c:1000:_crawl_directory] 0-gv3-replicate-0: crawling INDEX cfb7526b-cb8e-4c9a-90c1-0009a48b9d27
>[2013-03-07 17:00:46.993476] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.993494] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.993548] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.993848] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x50x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.993877] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=., count=0
>[2013-03-07 17:00:46.993891] T [syncop.c:608:syncop_readdir_cbk] 0-gv3-replicate-0: adding entry=.., count=1
>[2013-03-07 17:00:46.993941] T [rpc-clnt.c:1301:rpc_clnt_record] 0-gv3-client-1: Auth Info: pid: 4294967295, uid: 0, gid: 0, owner: b4d2dc7ce87f0000
>[2013-03-07 17:00:46.993965] T [rpc-clnt.c:1181:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 108, payload: 40, rpc hdr: 68
>[2013-03-07 17:00:46.994008] T [rpc-clnt.c:1498:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) to rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.994250] T [rpc-clnt.c:669:rpc_clnt_reply_init] 0-gv3-client-1: received rpc message (RPC XID: 0x51x Program: GlusterFS 3.3, ProgVers: 330, Proc: 28) from rpc-transport (gv3-client-1)
>[2013-03-07 17:00:46.994338] D [afr-self-heald.c:1138:afr_dir_crawl] 0-gv3-replicate-0: Crawl completed on gv3-client-1
>[2013-03-07 17:00:49.885368] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-glusterfs: breaking reconnect chain
>[2013-03-07 17:00:50.577967] T [socket.c:790:__socket_keepalive] 0-glusterfs: Keep-alive enabled for socket 15, interval 2, idle: 20
>[2013-03-07 17:00:50.885544] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv13-client-0: breaking reconnect chain
>[2013-03-07 17:00:50.885587] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv13-client-1: breaking reconnect chain
>[2013-03-07 17:00:50.885602] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv3-client-0: breaking reconnect chain
>[2013-03-07 17:00:50.885614] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv14-client-1: breaking reconnect chain
>[2013-03-07 17:00:50.885626] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv3-client-1: breaking reconnect chain
>[2013-03-07 17:00:50.885638] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv3-client-2: breaking reconnect chain
>[2013-03-07 17:00:50.885649] T [rpc-clnt.c:431:rpc_clnt_reconnect] 0-gv14-client-0: breaking reconnect chain

Actions: View

Attachments on bug 918437: 706248 | 706770