Attachment 578632 Details for Bug 802779 – nfs log

nfs log

nfs.log (text/x-log), 290.52 KB, created by M S Vishwanath Bhat on 2012-04-19 12:39:18 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: M S Vishwanath Bhat

Created: 2012-04-19 12:39:18 UTC

Size: 290.52 KB

patch

obsolete

>[2012-04-19 05:28:46.081213] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa35
>[2012-04-19 05:28:46.347181] I [nfs.c:795:init] 0-nfs: NFS service started
>[2012-04-19 05:28:46.367543] W [graph.c:325:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>[2012-04-19 05:28:46.367600] W [graph.c:325:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>[2012-04-19 05:28:46.367614] W [graph.c:325:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized
>[2012-04-19 05:28:46.367679] I [client.c:2151:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2012-04-19 05:28:46.370387] I [client.c:2151:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2012-04-19 05:28:46.372788] I [client.c:2151:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2012-04-19 05:28:46.375250] I [client.c:2151:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.63
>  4:     option remote-subvolume /data/bricks/hosdu_brick1
>  5:     option transport-type tcp
>  6:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
>  7:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.66
> 13:     option remote-subvolume /data/bricks/hosdu_brick2
> 14:     option transport-type tcp
> 15:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
> 16:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
> 17: end-volume
> 18: 
> 19: volume hosdu-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.65
> 22:     option remote-subvolume /data/bricks/hosdu_brick3
> 23:     option transport-type tcp
> 24:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
> 25:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
> 26: end-volume
> 27: 
> 28: volume hosdu-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.64
> 31:     option remote-subvolume /data/bricks/hosdu_brick4
> 32:     option transport-type tcp
> 33:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
> 34:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
> 35: end-volume
> 36: 
> 37: volume hosdu-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes hosdu-client-0 hosdu-client-1
> 40: end-volume
> 41: 
> 42: volume hosdu-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes hosdu-client-2 hosdu-client-3
> 45: end-volume
> 46: 
> 47: volume hosdu-dht
> 48:     type cluster/distribute
> 49:     subvolumes hosdu-replicate-0 hosdu-replicate-1
> 50: end-volume
> 51: 
> 52: volume hosdu
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes hosdu-dht
> 57: end-volume
> 58: 
> 59: volume nfs-server
> 60:     type nfs/server
> 61:     option nfs.dynamic-volumes on
> 62:     option nfs.nlm on
> 63:     option rpc-auth.addr.hosdu.allow *
> 64:     option nfs3.hosdu.volume-id 32e79cc3-a3ea-473a-9d00-68fc18272980
> 65:     subvolumes hosdu
> 66: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-19 05:28:46.378197] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 24009 (from 0)
>[2012-04-19 05:28:46.791489] E [client-handshake.c:1713:client_query_portmap_cbk] 0-hosdu-client-2: failed to get the port number for remote subvolume
>[2012-04-19 05:28:46.791550] I [client.c:2099:client_rpc_notify] 0-hosdu-client-2: disconnected
>[2012-04-19 05:28:46.802314] E [client-handshake.c:1713:client_query_portmap_cbk] 0-hosdu-client-3: failed to get the port number for remote subvolume
>[2012-04-19 05:28:46.802373] I [client.c:2099:client_rpc_notify] 0-hosdu-client-3: disconnected
>[2012-04-19 05:28:46.802401] E [afr-common.c:3577:afr_notify] 0-hosdu-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-04-19 05:28:47.215980] E [client-handshake.c:1713:client_query_portmap_cbk] 0-hosdu-client-1: failed to get the port number for remote subvolume
>[2012-04-19 05:28:47.216037] I [client.c:2099:client_rpc_notify] 0-hosdu-client-1: disconnected
>[2012-04-19 05:28:49.912389] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-19 05:28:49.912421] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-19 05:28:50.103602] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 05:28:50.112582] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 24009 (from 0)
>[2012-04-19 05:28:50.116386] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 24009 (from 0)
>[2012-04-19 05:28:50.121197] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-0: Connected to 172.17.251.63:24009, attached to remote volume '/data/bricks/hosdu_brick1'.
>[2012-04-19 05:28:50.121261] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 05:28:50.122495] I [afr-common.c:3540:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2012-04-19 05:28:50.122996] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2012-04-19 05:28:50.124102] I [afr-common.c:1882:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2012-04-19 05:28:50.124422] I [dht-layout.c:600:dht_layout_normalize] 0-hosdu-dht: found anomalies in /. holes=1 overlaps=0
>[2012-04-19 05:28:50.124445] W [dht-selfheal.c:881:dht_selfheal_directory] 0-hosdu-dht: 1 subvolumes down -- not fixing
>[2012-04-19 05:28:51.121014] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 24009 (from 0)
>[2012-04-19 05:28:53.123121] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 05:28:53.123454] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-2: Connected to 172.17.251.65:24009, attached to remote volume '/data/bricks/hosdu_brick3'.
>[2012-04-19 05:28:53.123472] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 05:28:53.123517] I [afr-common.c:3540:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online.
>[2012-04-19 05:28:53.123753] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2012-04-19 05:28:53.126116] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 05:28:53.126572] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-3: Connected to 172.17.251.64:24009, attached to remote volume '/data/bricks/hosdu_brick4'.
>[2012-04-19 05:28:53.126603] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 05:28:53.126912] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2012-04-19 05:28:54.131205] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 05:28:54.131884] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-1: Connected to 172.17.251.66:24009, attached to remote volume '/data/bricks/hosdu_brick2'.
>[2012-04-19 05:28:54.131915] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 05:28:54.132390] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2012-04-19 05:32:19.626985] I [afr-common.c:1882:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2012-04-19 05:34:53.067617] E [nlm4.c:508:nsm_monitor] 0-nfs-NLM: clnt_call(): RPC: Success
>[2012-04-19 05:44:59.932281] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 345b1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:00.008342] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: ea5b1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:00.026105] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 135c1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:00.050688] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 3d5c1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:00.685395] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 3e621901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:01.025142] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 93651901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:01.093754] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 36661901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:01.458025] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: a2691901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:01.541835] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 5f6a1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:01.635378] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 486b1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:01.912322] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: ca6d1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:02.236321] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: a711901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:02.478370] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 4c731901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:02.816768] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 69751901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:02.874163] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: a9751901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:02.937162] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: fa751901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:03.128033] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: ed761901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:03.569855] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 287a1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:03.936619] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 667d1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.148734] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 197f1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.187265] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 587f1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.285679] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 1b801901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.494373] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: ad811901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.590642] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 6c821901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.624729] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: b1821901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.732412] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 6d831901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:04.768753] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: d2831901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.115250] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 51861901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.162408] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 8e861901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.177672] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: ab861901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.416468] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 22881901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.433552] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 47881901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.450030] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 65881901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.524752] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: a7881901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.642226] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 5b891901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.778442] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 408a1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.826995] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 9f8a1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.869639] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: d58a1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:05.935678] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 358b1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.018932] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: cd8b1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.064436] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 148c1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.085664] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 308c1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.205712] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: d28c1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.211510] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:04f61563-fc74-47fc-908f-ac6a569a7f06>: Invalid argument
>[2012-04-19 05:45:06.211679] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 04f61563-fc74-47fc-908f-ac6a569a7f06
>[2012-04-19 05:45:06.211696] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: e48c1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.215100] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:e7f7b768-24f1-434d-b1c2-6637f6ddc621>: Invalid argument
>[2012-04-19 05:45:06.215699] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : e7f7b768-24f1-434d-b1c2-6637f6ddc621
>[2012-04-19 05:45:06.215717] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: ec8c1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.224201] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : ace63875-7e41-4dcd-a979-33f330e9839b
>[2012-04-19 05:45:06.226569] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:a029d0a4-c780-43e3-a616-14e927917713>: Invalid argument
>[2012-04-19 05:45:06.226595] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : a029d0a4-c780-43e3-a616-14e927917713
>[2012-04-19 05:45:06.226643] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: f58c1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.226751] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: ef8c1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.230496] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : ace63875-7e41-4dcd-a979-33f330e9839b
>[2012-04-19 05:45:06.233221] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:7cf6e4cf-4f7b-4542-8461-a98cb8521f22>: Invalid argument
>[2012-04-19 05:45:06.233334] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 7cf6e4cf-4f7b-4542-8461-a98cb8521f22
>[2012-04-19 05:45:06.233350] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: fc8c1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.240496] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : ace63875-7e41-4dcd-a979-33f330e9839b
>[2012-04-19 05:45:06.246387] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : ace63875-7e41-4dcd-a979-33f330e9839b
>[2012-04-19 05:45:06.252728] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 148d1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.324953] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 5b8d1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.329338] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b60383c2-d9fd-4a78-ac41-afe5d58ad52e>: Invalid argument
>[2012-04-19 05:45:06.329382] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : b60383c2-d9fd-4a78-ac41-afe5d58ad52e
>[2012-04-19 05:45:06.329397] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 668d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.332931] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:862d7185-54a4-4077-8860-c62b880588a4>: Invalid argument
>[2012-04-19 05:45:06.332968] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 862d7185-54a4-4077-8860-c62b880588a4
>[2012-04-19 05:45:06.332994] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 6b8d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.336724] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 05c49187-0e85-454c-96a8-a3893dc8e900
>[2012-04-19 05:45:06.337614] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:3ec2763c-3882-4305-88e9-c21dad37a761>: Invalid argument
>[2012-04-19 05:45:06.337638] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 3ec2763c-3882-4305-88e9-c21dad37a761
>[2012-04-19 05:45:06.337650] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 728d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.342408] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 05c49187-0e85-454c-96a8-a3893dc8e900
>[2012-04-19 05:45:06.343472] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:1de716a7-1f48-435a-bde1-41e8c03f5894>: Invalid argument
>[2012-04-19 05:45:06.343497] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 1de716a7-1f48-435a-bde1-41e8c03f5894
>[2012-04-19 05:45:06.343509] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 798d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.347627] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 05c49187-0e85-454c-96a8-a3893dc8e900
>[2012-04-19 05:45:06.347842] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 05c49187-0e85-454c-96a8-a3893dc8e900
>[2012-04-19 05:45:06.351960] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 788d1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.355227] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:d434a061-4b81-4b8e-81a8-886112a17fbd>: Invalid argument
>[2012-04-19 05:45:06.355252] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : d434a061-4b81-4b8e-81a8-886112a17fbd
>[2012-04-19 05:45:06.355264] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 808d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.362071] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:422edc3e-d001-4e2e-9dc4-210e44d0abd3>: Invalid argument
>[2012-04-19 05:45:06.362096] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 422edc3e-d001-4e2e-9dc4-210e44d0abd3
>[2012-04-19 05:45:06.362109] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 818d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.363964] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : c2e948b2-7f3a-45b5-9e1f-2a9d99d7f050
>[2012-04-19 05:45:06.365653] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:5487c6e6-e2af-4074-baff-76f1a1519e52>: Invalid argument
>[2012-04-19 05:45:06.365718] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 5487c6e6-e2af-4074-baff-76f1a1519e52
>[2012-04-19 05:45:06.365738] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 888d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.365916] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : c2e948b2-7f3a-45b5-9e1f-2a9d99d7f050
>[2012-04-19 05:45:06.368166] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:1393dda4-4eec-4e35-800e-8f777b0aeec3>: Invalid argument
>[2012-04-19 05:45:06.368214] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 1393dda4-4eec-4e35-800e-8f777b0aeec3
>[2012-04-19 05:45:06.368249] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 8f8d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.373698] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : c2e948b2-7f3a-45b5-9e1f-2a9d99d7f050
>[2012-04-19 05:45:06.374685] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:79024ccb-96a9-4c4a-9313-85248c14d042>: Invalid argument
>[2012-04-19 05:45:06.374882] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 79024ccb-96a9-4c4a-9313-85248c14d042
>[2012-04-19 05:45:06.375035] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 948d1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.377451] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : c2e948b2-7f3a-45b5-9e1f-2a9d99d7f050
>[2012-04-19 05:45:06.378565] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : c2e948b2-7f3a-45b5-9e1f-2a9d99d7f050
>[2012-04-19 05:45:06.445228] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: e18d1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.510969] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 1c8e1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.639265] W [nfs3.c:3529:nfs3svc_rmdir_cbk] 0-nfs: 878e1901: /clients => -1 (Directory not empty)
>[2012-04-19 05:45:06.640259] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:ccfe3c55-42d3-4f7c-a2e1-b6b67da28363>: Invalid argument
>[2012-04-19 05:45:06.640285] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : ccfe3c55-42d3-4f7c-a2e1-b6b67da28363
>[2012-04-19 05:45:06.640298] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 888e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.642170] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:523309fa-1dda-4492-bb5d-bfe3c854ea92>: Invalid argument
>[2012-04-19 05:45:06.642195] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 523309fa-1dda-4492-bb5d-bfe3c854ea92
>[2012-04-19 05:45:06.642207] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 8b8e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.643072] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.645722] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.646859] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:22c5869e-e813-41b7-a6de-b19d809851ec>: Invalid argument
>[2012-04-19 05:45:06.646886] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 22c5869e-e813-41b7-a6de-b19d809851ec
>[2012-04-19 05:45:06.646899] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 908e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.650102] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:d8e71200-1f12-4c7d-94ae-b065af642849>: Invalid argument
>[2012-04-19 05:45:06.650128] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : d8e71200-1f12-4c7d-94ae-b065af642849
>[2012-04-19 05:45:06.650141] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 938e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.650426] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.652455] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:10e83e22-06af-4b87-8469-43fcf48ba360>: Invalid argument
>[2012-04-19 05:45:06.652480] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 10e83e22-06af-4b87-8469-43fcf48ba360
>[2012-04-19 05:45:06.652493] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 958e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.653559] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.654343] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:22e5668b-b8ba-450f-be66-16451bfd8af5>: Invalid argument
>[2012-04-19 05:45:06.654372] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 22e5668b-b8ba-450f-be66-16451bfd8af5
>[2012-04-19 05:45:06.654391] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 968e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.657417] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:2295f132-b651-442f-aee7-7752c48485f7>: Invalid argument
>[2012-04-19 05:45:06.657442] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 2295f132-b651-442f-aee7-7752c48485f7
>[2012-04-19 05:45:06.657455] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 998e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.657686] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.660674] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.661095] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:7de2812e-bb2e-41ec-98af-d7afebc956e4>: Invalid argument
>[2012-04-19 05:45:06.661708] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 7de2812e-bb2e-41ec-98af-d7afebc956e4
>[2012-04-19 05:45:06.661983] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 9b8e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.664594] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.666191] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:c9c2eba3-ae49-47c4-b729-cb2bd2921154>: Invalid argument
>[2012-04-19 05:45:06.666224] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : c9c2eba3-ae49-47c4-b729-cb2bd2921154
>[2012-04-19 05:45:06.666239] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: 9e8e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.668759] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.669215] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:4b89ded7-bfa2-45e3-b971-dd9195983a3a>: Invalid argument
>[2012-04-19 05:45:06.669239] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 4b89ded7-bfa2-45e3-b971-dd9195983a3a
>[2012-04-19 05:45:06.669252] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: a18e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.671694] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.671812] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:3a7ef63b-bc3b-4482-8caa-90afcf7ea348>: Invalid argument
>[2012-04-19 05:45:06.671834] E [nfs3.c:753:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 3a7ef63b-bc3b-4482-8caa-90afcf7ea348
>[2012-04-19 05:45:06.671855] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: a38e1901, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
>[2012-04-19 05:45:06.673318] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:45:06.674388] E [nfs3.c:3412:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve FH: (172.17.251.62:795) hosdu : 015fd863-44c1-44e2-ab98-c5cbd355f1f2
>[2012-04-19 05:52:53.074665] W [socket.c:1521:__socket_proto_state_machine] 0-NLM-client: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.62:39083)
>[2012-04-19 05:52:53.074851] I [mem-pool.c:585:mem_pool_destroy] 0-nfs-server: size=2236 max=2 total=562
>[2012-04-19 05:52:53.074888] I [mem-pool.c:585:mem_pool_destroy] 0-nfs-server: size=124 max=2 total=562
>[2012-04-19 06:50:25.498324] E [nlm4.c:508:nsm_monitor] 0-nfs-NLM: clnt_call(): RPC: Success
>[2012-04-19 06:50:38.103197] W [socket.c:204:__socket_rwv] 0-hosdu-client-0: readv failed (Connection reset by peer)
>[2012-04-19 06:50:38.103241] W [socket.c:1521:__socket_proto_state_machine] 0-hosdu-client-0: reading from socket failed. Error (Connection reset by peer), peer (172.17.251.63:24009)
>[2012-04-19 06:50:38.122729] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-19 06:50:38.080682 (xid=0x1485402x)
>[2012-04-19 06:50:38.122808] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.122884] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-04-19 06:50:38.080476 (xid=0x1485401x)
>[2012-04-19 06:50:38.122911] W [client3_1-fops.c:2610:client3_1_lookup_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected. Path: /clients/client3/~dmtmp/WORDPRO/BENCHS.LWP
>[2012-04-19 06:50:38.123060] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(ACCESS(22)) called at 2012-04-19 06:50:38.080747 (xid=0x1485403x)
>[2012-04-19 06:50:38.123104] W [client3_1-fops.c:1326:client3_1_access_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.123202] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-04-19 06:50:38.080942 (xid=0x1485404x)
>[2012-04-19 06:50:38.123232] W [client3_1-fops.c:1773:client3_1_fxattrop_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.128481] I [socket.c:2314:socket_submit_request] 0-hosdu-client-0: not connected (priv->connected = 0)
>[2012-04-19 06:50:38.128519] W [rpc-clnt.c:1507:rpc_clnt_submit] 0-hosdu-client-0: failed to submit rpc-request (XID: 0x1485410x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (hosdu-client-0)
>[2012-04-19 06:50:38.128543] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.128674] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-04-19 06:50:38.081042 (xid=0x1485405x)
>[2012-04-19 06:50:38.128706] W [client3_1-fops.c:2610:client3_1_lookup_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected. Path: /clients/client45/~dmtmp/EXCEL/BEED0000
>[2012-04-19 06:50:38.128756] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2012-04-19 06:50:38.081560 (xid=0x1485406x)
>[2012-04-19 06:50:38.128832] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.128920] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-04-19 06:50:38.081867 (xid=0x1485407x)
>[2012-04-19 06:50:38.129013] W [client3_1-fops.c:2680:client3_1_readv_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.129112] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-19 06:50:38.082203 (xid=0x1485408x)
>[2012-04-19 06:50:38.129142] W [client3_1-fops.c:822:client3_1_writev_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.129189] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-0: forced unwinding frame type(GlusterFS 3.1) op(FSTAT(25)) called at 2012-04-19 06:50:38.082667 (xid=0x1485409x)
>[2012-04-19 06:50:38.129222] W [client3_1-fops.c:1438:client3_1_fstat_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.129261] I [client.c:2099:client_rpc_notify] 0-hosdu-client-0: disconnected
>[2012-04-19 06:50:38.129881] W [client3_1-fops.c:1773:client3_1_fxattrop_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.130252] E [socket.c:1724:socket_connect_finish] 0-hosdu-client-0: connection to 172.17.251.63:24009 failed (Connection refused)
>[2012-04-19 06:50:38.133370] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.188338] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.193649] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.235181] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.239383] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.680201] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:38.698341] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.211323] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.217036] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.348210] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.350330] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.473986] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.477332] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.518154] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.519987] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.532966] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.535708] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.549111] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.550744] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.809061] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.811280] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.821040] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.823458] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.943658] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.946310] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.967056] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:39.976218] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:40.338666] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:40.340264] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:40.544326] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:40.547567] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.206055] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.209688] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.467491] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.469593] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.495448] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.496253] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.653498] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.654881] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.763440] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.764836] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.841109] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.842120] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.905606] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:41.907599] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:42.086764] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:42.090761] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:42.478147] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:42.485506] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:42.887854] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:42.890800] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:44.157371] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:44.161385] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:44.517692] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:44.519325] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:44.907348] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:44.913923] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:45.366041] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:45.368504] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:45.535016] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:45.540833] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:45.938338] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:45.942693] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:46.762052] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:46.765291] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:47.128241] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:50:47.131630] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:50.143289] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:50.149222] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:50.334829] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:50.339664] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:57.114099] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:57.115488] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:57.630800] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:57.632254] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.595931] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.597915] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.766096] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.773804] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.780672] W [socket.c:204:__socket_rwv] 0-hosdu-client-3: readv failed (Connection reset by peer)
>[2012-04-19 06:51:58.780700] W [socket.c:1521:__socket_proto_state_machine] 0-hosdu-client-3: reading from socket failed. Error (Connection reset by peer), peer (172.17.251.64:24009)
>[2012-04-19 06:51:58.780799] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2012-04-19 06:51:58.775176 (xid=0x1527229x)
>[2012-04-19 06:51:58.780823] W [client3_1-fops.c:1589:client3_1_entrylk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.780887] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2012-04-19 06:51:58.778514 (xid=0x1527235x)
>[2012-04-19 06:51:58.780905] W [client3_1-fops.c:1589:client3_1_entrylk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.780944] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-19 06:51:58.778840 (xid=0x1527236x)
>[2012-04-19 06:51:58.780958] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.780996] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-19 06:51:58.778997 (xid=0x1527237x)
>[2012-04-19 06:51:58.781011] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.781089] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2012-04-19 06:51:58.773452 (xid=0x1527226x)
>[2012-04-19 06:51:58.781105] W [client3_1-fops.c:764:client3_1_statfs_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.781172] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(FSTAT(25)) called at 2012-04-19 06:51:58.774193 (xid=0x1527227x)
>[2012-04-19 06:51:58.781196] W [client3_1-fops.c:1438:client3_1_fstat_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.781236] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2012-04-19 06:51:58.774406 (xid=0x1527228x)
>[2012-04-19 06:51:58.781250] W [client3_1-fops.c:475:client3_1_stat_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.781286] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(FSTAT(25)) called at 2012-04-19 06:51:58.775478 (xid=0x1527230x)
>[2012-04-19 06:51:58.781376] W [client3_1-fops.c:1438:client3_1_fstat_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.781449] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-04-19 06:51:58.775761 (xid=0x1527231x)
>[2012-04-19 06:51:58.781470] W [client3_1-fops.c:1773:client3_1_fxattrop_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.784155] I [socket.c:2314:socket_submit_request] 0-hosdu-client-3: not connected (priv->connected = 0)
>[2012-04-19 06:51:58.784184] W [rpc-clnt.c:1507:rpc_clnt_submit] 0-hosdu-client-3: failed to submit rpc-request (XID: 0x1527238x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (hosdu-client-3)
>[2012-04-19 06:51:58.784199] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.784260] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2012-04-19 06:51:58.775898 (xid=0x1527232x)
>[2012-04-19 06:51:58.784280] W [client3_1-fops.c:2299:client3_1_readdirp_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.784313] W [rpc-clnt.c:1507:rpc_clnt_submit] 0-hosdu-client-3: failed to submit rpc-request (XID: 0x1527239x Program: GlusterFS 3.1, ProgVers: 330, Proc: 25) to rpc-transport (hosdu-client-3)
>[2012-04-19 06:51:58.784329] W [client3_1-fops.c:1438:client3_1_fstat_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.784413] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2012-04-19 06:51:58.776108 (xid=0x1527233x)
>[2012-04-19 06:51:58.784431] W [client3_1-fops.c:764:client3_1_statfs_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.784541] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7fe3d4a72c98] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7fe3d4a72950] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fe3d4a723be]))) 0-hosdu-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-04-19 06:51:58.776696 (xid=0x1527234x)
>[2012-04-19 06:51:58.784560] W [client3_1-fops.c:2610:client3_1_lookup_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected. Path: /clients/client20/~dmtmp/WORDPRO/BENCHS1A.PRN
>[2012-04-19 06:51:58.784661] I [client.c:2099:client_rpc_notify] 0-hosdu-client-3: disconnected
>[2012-04-19 06:51:58.784722] E [socket.c:1724:socket_connect_finish] 0-hosdu-client-3: connection to 172.17.251.64:24009 failed (Connection refused)
>[2012-04-19 06:51:58.785514] W [client3_1-fops.c:1589:client3_1_entrylk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.785600] W [client3_1-fops.c:1589:client3_1_entrylk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.785655] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:58.785743] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.105211] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.107113] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.193721] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.195943] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.222374] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.223527] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.224328] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.227508] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.301513] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.303202] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.340219] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.341276] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.528270] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.529827] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.607490] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.611635] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.708754] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.711361] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.732921] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.733746] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.757801] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.758848] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.771302] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.775812] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.819005] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.826546] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.830117] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.835082] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.886388] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:51:59.889393] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.099068] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.105923] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.213250] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.222962] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.536993] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.541545] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.716941] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.721752] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.885295] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.889302] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.940900] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:00.946544] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.129190] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.132685] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.171858] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.176243] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.254689] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.256183] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.421819] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.425102] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.478952] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.481921] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.577188] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.581232] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.846717] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:01.849041] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.105689] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.107925] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.335515] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.336505] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.447978] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.450004] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.905080] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:02.906742] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:03.276913] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:03.281155] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:03.884103] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:03.890092] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.082097] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.085231] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.213447] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.214190] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.245376] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.248228] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.638150] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.642424] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.714258] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:04.716912] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:06.781551] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:06.784006] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.059420] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.060900] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.121390] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.122701] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.146627] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.150928] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.214682] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.216699] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.289335] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.292586] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.524650] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.527567] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.656985] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.658162] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.676907] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:07.678796] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.039407] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.042727] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.096432] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.098244] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.170701] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.171639] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.390847] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.392068] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.590226] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.593889] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.666436] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.667305] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.887581] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:08.888880] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.007979] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.014378] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.416072] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.417185] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.418347] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.419370] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.676980] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:09.679174] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:10.034987] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:10.039843] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:10.643375] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:10.645012] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:10.936423] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:10.938694] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.011557] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.013145] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.262005] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.263299] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.429897] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.430984] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.730850] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.731528] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.938757] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:11.939321] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:12.410196] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:12.413423] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:12.567002] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:12.571140] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:12.839393] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:12.845110] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.172359] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.176422] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.180089] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.183419] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.194922] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.200032] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.264638] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.269986] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.411654] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:13.414215] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-3: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:18.849086] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:18.850358] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:19.433304] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:19.434540] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:25.654002] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:25.656707] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:26.347095] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:26.348505] W [client3_1-fops.c:2173:client3_1_lk_cbk] 0-hosdu-client-0: remote operation failed: Transport endpoint is not connected
>[2012-04-19 06:52:54.013809] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa35
>[2012-04-19 06:52:55.238175] I [nfs.c:795:init] 0-nfs: NFS service started
>[2012-04-19 06:52:55.248608] W [graph.c:325:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>[2012-04-19 06:52:55.248649] W [graph.c:325:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>[2012-04-19 06:52:55.248673] W [graph.c:325:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized
>[2012-04-19 06:52:55.248718] I [client.c:2151:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2012-04-19 06:52:55.251330] I [client.c:2151:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2012-04-19 06:52:55.253648] I [client.c:2151:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2012-04-19 06:52:55.256197] I [client.c:2151:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.63
>  4:     option remote-subvolume /data/bricks/hosdu_brick1
>  5:     option transport-type tcp
>  6:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
>  7:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.66
> 13:     option remote-subvolume /data/bricks/hosdu_brick2
> 14:     option transport-type tcp
> 15:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
> 16:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
> 17: end-volume
> 18: 
> 19: volume hosdu-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.65
> 22:     option remote-subvolume /data/bricks/hosdu_brick3
> 23:     option transport-type tcp
> 24:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
> 25:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
> 26: end-volume
> 27: 
> 28: volume hosdu-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.64
> 31:     option remote-subvolume /data/bricks/hosdu_brick4
> 32:     option transport-type tcp
> 33:     option username b2cd48bb-e0a3-499d-99cf-2fee82e62887
> 34:     option password 66bd2d62-2941-404e-a486-ee017c230ae6
> 35: end-volume
> 36: 
> 37: volume hosdu-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes hosdu-client-0 hosdu-client-1
> 40: end-volume
> 41: 
> 42: volume hosdu-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes hosdu-client-2 hosdu-client-3
> 45: end-volume
> 46: 
> 47: volume hosdu-dht
> 48:     type cluster/distribute
> 49:     subvolumes hosdu-replicate-0 hosdu-replicate-1
> 50: end-volume
> 51: 
> 52: volume hosdu
> 53:     type debug/io-stats
> 54:     option latency-measurement off
> 55:     option count-fop-hits off
> 56:     subvolumes hosdu-dht
> 57: end-volume
> 58: 
> 59: volume nfs-server
> 60:     type nfs/server
> 61:     option nfs.dynamic-volumes on
> 62:     option nfs.nlm on
> 63:     option rpc-auth.addr.hosdu.allow *
> 64:     option nfs3.hosdu.volume-id 32e79cc3-a3ea-473a-9d00-68fc18272980
> 65:     subvolumes hosdu
> 66: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-19 06:52:55.259223] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 24009 (from 0)
>[2012-04-19 06:52:55.259644] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 24009 (from 0)
>[2012-04-19 06:52:55.259726] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 24009 (from 0)
>[2012-04-19 06:52:55.259834] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 24009 (from 0)
>[2012-04-19 06:52:56.767832] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-19 06:52:56.767876] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-19 06:52:57.767990] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 16
>[2012-04-19 06:52:57.768060] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-19 06:52:59.032941] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 06:52:59.033211] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-0: Connected to 172.17.251.63:24009, attached to remote volume '/data/bricks/hosdu_brick1'.
>[2012-04-19 06:52:59.033230] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 06:52:59.033759] I [afr-common.c:3540:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2012-04-19 06:52:59.033825] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 06:52:59.033885] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2012-04-19 06:52:59.034152] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-3: Connected to 172.17.251.64:24009, attached to remote volume '/data/bricks/hosdu_brick4'.
>[2012-04-19 06:52:59.034169] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 06:52:59.034210] I [afr-common.c:3540:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-3' came back up; going online.
>[2012-04-19 06:52:59.034369] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2012-04-19 06:52:59.036177] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 06:52:59.036552] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-2: Connected to 172.17.251.65:24009, attached to remote volume '/data/bricks/hosdu_brick3'.
>[2012-04-19 06:52:59.036570] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 06:52:59.036866] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2012-04-19 06:52:59.039230] I [client-handshake.c:1632:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3.0qa35, Num (1298437), Version (330)
>[2012-04-19 06:52:59.039664] I [client-handshake.c:1429:client_setvolume_cbk] 0-hosdu-client-1: Connected to 172.17.251.66:24009, attached to remote volume '/data/bricks/hosdu_brick2'.
>[2012-04-19 06:52:59.039684] I [client-handshake.c:1441:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-19 06:52:59.039954] I [client-handshake.c:456:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2012-04-19 06:52:59.040452] I [afr-common.c:1882:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2012-04-19 06:52:59.041030] I [afr-common.c:1882:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2012-04-19 06:53:02.215693] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.216581] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:35145a3c-80e5-493b-8504-1a32d1cc68ed>.
>[2012-04-19 06:53:02.216604] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:35145a3c-80e5-493b-8504-1a32d1cc68ed>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.217329] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:1edaaf99-4a2b-4611-94f2-c58b9b90b499>.
>[2012-04-19 06:53:02.217352] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:1edaaf99-4a2b-4611-94f2-c58b9b90b499>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.218185] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:3e14b2d4-088c-4912-93b2-76fc1dad4cb8>.
>[2012-04-19 06:53:02.218208] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:3e14b2d4-088c-4912-93b2-76fc1dad4cb8>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.219015] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.219853] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.220372] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.221945] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.223821] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.224406] I [afr-self-heal-entry.c:2330:afr_sh_entry_fix] 0-hosdu-replicate-0: <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>: Performing conservative merge
>[2012-04-19 06:53:02.225166] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:65499379-c2ac-4013-af6e-ba9f27fb2d05>.
>[2012-04-19 06:53:02.225188] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:65499379-c2ac-4013-af6e-ba9f27fb2d05>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.225391] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:35145a3c-80e5-493b-8504-1a32d1cc68ed>.
>[2012-04-19 06:53:02.225410] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:35145a3c-80e5-493b-8504-1a32d1cc68ed>.
>[2012-04-19 06:53:02.225666] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.225992] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.226259] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:1edaaf99-4a2b-4611-94f2-c58b9b90b499>.
>[2012-04-19 06:53:02.226280] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:1edaaf99-4a2b-4611-94f2-c58b9b90b499>.
>[2012-04-19 06:53:02.226990] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:caeacc73-2956-4210-8166-3ed512edb549>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.227170] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:a22d5213-e34b-40cb-b757-12f096e97680>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.227342] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.227369] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:3e14b2d4-088c-4912-93b2-76fc1dad4cb8>.
>[2012-04-19 06:53:02.227416] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:3e14b2d4-088c-4912-93b2-76fc1dad4cb8>.
>[2012-04-19 06:53:02.227660] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:35145a3c-80e5-493b-8504-1a32d1cc68ed>
>[2012-04-19 06:53:02.227682] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:35145a3c-80e5-493b-8504-1a32d1cc68ed> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:02.227722] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.228201] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:1edaaf99-4a2b-4611-94f2-c58b9b90b499>
>[2012-04-19 06:53:02.228223] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:1edaaf99-4a2b-4611-94f2-c58b9b90b499> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:02.228813] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:35145a3c-80e5-493b-8504-1a32d1cc68ed>
>[2012-04-19 06:53:02.229099] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:3e14b2d4-088c-4912-93b2-76fc1dad4cb8>
>[2012-04-19 06:53:02.229119] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:3e14b2d4-088c-4912-93b2-76fc1dad4cb8> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:02.229699] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:1edaaf99-4a2b-4611-94f2-c58b9b90b499>
>[2012-04-19 06:53:02.230921] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.231242] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:3e14b2d4-088c-4912-93b2-76fc1dad4cb8>
>[2012-04-19 06:53:02.231750] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.234426] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:65499379-c2ac-4013-af6e-ba9f27fb2d05>.
>[2012-04-19 06:53:02.234449] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:65499379-c2ac-4013-af6e-ba9f27fb2d05>.
>[2012-04-19 06:53:02.234679] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.235518] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:65499379-c2ac-4013-af6e-ba9f27fb2d05>
>[2012-04-19 06:53:02.235540] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:65499379-c2ac-4013-af6e-ba9f27fb2d05> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:02.236759] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:65499379-c2ac-4013-af6e-ba9f27fb2d05>
>[2012-04-19 06:53:02.237230] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.237703] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.241075] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:02.242461] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.244902] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.246078] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.246185] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.246672] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.246808] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.247281] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.247364] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>/HANDOUT.DOC on hosdu-client-1
>[2012-04-19 06:53:02.247935] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>/TIPS.DOC on hosdu-client-1
>[2012-04-19 06:53:02.248734] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.249296] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.249625] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:caeacc73-2956-4210-8166-3ed512edb549>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.250010] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:a22d5213-e34b-40cb-b757-12f096e97680>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.250127] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.250417] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:02.253407] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWPCB.PPT on hosdu-client-1
>[2012-04-19 06:53:02.254216] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:b3682e4f-38c3-4114-b6f9-03ec6e1be003>.
>[2012-04-19 06:53:02.254238] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:b3682e4f-38c3-4114-b6f9-03ec6e1be003>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.256138] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.258916] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.258954] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:a22d5213-e34b-40cb-b757-12f096e97680>/NEWTIPS.PPT on hosdu-client-1
>[2012-04-19 06:53:02.262627] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:02.285510] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.286049] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL1146.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.286801] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.287841] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:e9e2b5c0-d81b-4c14-9db1-2200caad1167>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.288534] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:e9e2b5c0-d81b-4c14-9db1-2200caad1167>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.289299] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.289397] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWPCB.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.289690] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:a22d5213-e34b-40cb-b757-12f096e97680>/NEWTIPS.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.289733] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:e9e2b5c0-d81b-4c14-9db1-2200caad1167>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.291038] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.341861] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.345533] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.377549] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>
>[2012-04-19 06:53:02.377649] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>
>[2012-04-19 06:53:02.391705] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>
>[2012-04-19 06:53:02.393302] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>
>[2012-04-19 06:53:02.393565] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>
>[2012-04-19 06:53:02.393822] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:caeacc73-2956-4210-8166-3ed512edb549>
>[2012-04-19 06:53:02.394360] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:a22d5213-e34b-40cb-b757-12f096e97680>
>[2012-04-19 06:53:02.394434] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>
>[2012-04-19 06:53:02.412005] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:02.452546] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT, reason: lookup detected pending operations
>[2012-04-19 06:53:02.453836] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:bc601fe2-25c7-4308-af07-fd0a5063180b>/SEED/SMALL.FIL, reason: lookup detected pending operations
>[2012-04-19 06:53:02.463408] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>
>[2012-04-19 06:53:02.468912] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWPCB.PPT on hosdu-client-1
>[2012-04-19 06:53:02.474013] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWPCB.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.495173] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT
>[2012-04-19 06:53:02.496463] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT, reason: lookup detected pending operations
>[2012-04-19 06:53:02.506347] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:e9e2b5c0-d81b-4c14-9db1-2200caad1167>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:02.523842] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:e9e2b5c0-d81b-4c14-9db1-2200caad1167>
>[2012-04-19 06:53:02.525031] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:e9e2b5c0-d81b-4c14-9db1-2200caad1167>
>[2012-04-19 06:53:02.565664] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:e9e2b5c0-d81b-4c14-9db1-2200caad1167>
>[2012-04-19 06:53:02.575846] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:79aee3f6-93d5-4482-9917-22b22562457b>.
>[2012-04-19 06:53:02.575926] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:79aee3f6-93d5-4482-9917-22b22562457b>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.576592] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.579867] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:79aee3f6-93d5-4482-9917-22b22562457b>.
>[2012-04-19 06:53:02.579898] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:79aee3f6-93d5-4482-9917-22b22562457b>.
>[2012-04-19 06:53:02.580121] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.580407] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:23b8e730-33d9-4f6f-b4db-dab434cf74b4>.
>[2012-04-19 06:53:02.580429] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:23b8e730-33d9-4f6f-b4db-dab434cf74b4>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.580724] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.580846] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:79aee3f6-93d5-4482-9917-22b22562457b>
>[2012-04-19 06:53:02.580866] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:79aee3f6-93d5-4482-9917-22b22562457b> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:02.580911] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.581131] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.581371] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.581527] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:79aee3f6-93d5-4482-9917-22b22562457b>
>[2012-04-19 06:53:02.581847] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.584342] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.585188] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.586347] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>/HANDOUT.DOC on hosdu-client-1
>[2012-04-19 06:53:02.587406] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>/TIPS.DOC on hosdu-client-1
>[2012-04-19 06:53:02.589021] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWPCB.PPT on hosdu-client-1
>[2012-04-19 06:53:02.589908] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:032d1dee-5036-4872-b1b0-695b9b511e03>.
>[2012-04-19 06:53:02.589932] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:032d1dee-5036-4872-b1b0-695b9b511e03>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.590350] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.592242] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.592275] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.596424] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:032d1dee-5036-4872-b1b0-695b9b511e03>.
>[2012-04-19 06:53:02.596446] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:032d1dee-5036-4872-b1b0-695b9b511e03>.
>[2012-04-19 06:53:02.597459] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWPCB.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:02.597509] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:032d1dee-5036-4872-b1b0-695b9b511e03>
>[2012-04-19 06:53:02.597526] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:032d1dee-5036-4872-b1b0-695b9b511e03> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:02.600067] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:032d1dee-5036-4872-b1b0-695b9b511e03>
>[2012-04-19 06:53:02.600723] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.601788] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.602209] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>
>[2012-04-19 06:53:02.602534] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.603214] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>
>[2012-04-19 06:53:02.620809] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:624c2f0f-92fe-4de5-a71f-def2b4b4bde7>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.762524] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:b3682e4f-38c3-4114-b6f9-03ec6e1be003>.
>[2012-04-19 06:53:02.762572] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:b3682e4f-38c3-4114-b6f9-03ec6e1be003>.
>[2012-04-19 06:53:02.763350] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:23b8e730-33d9-4f6f-b4db-dab434cf74b4>.
>[2012-04-19 06:53:02.763401] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:23b8e730-33d9-4f6f-b4db-dab434cf74b4>.
>[2012-04-19 06:53:02.778196] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:b3682e4f-38c3-4114-b6f9-03ec6e1be003>
>[2012-04-19 06:53:02.778220] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:b3682e4f-38c3-4114-b6f9-03ec6e1be003> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:02.778717] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:23b8e730-33d9-4f6f-b4db-dab434cf74b4>
>[2012-04-19 06:53:02.778741] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:23b8e730-33d9-4f6f-b4db-dab434cf74b4> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:02.915001] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:b3682e4f-38c3-4114-b6f9-03ec6e1be003>
>[2012-04-19 06:53:02.916371] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.932111] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:23b8e730-33d9-4f6f-b4db-dab434cf74b4>
>[2012-04-19 06:53:02.932452] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.933509] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.935225] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:f9ce9cd7-92fa-4d32-b089-1ee7106178ee>.
>[2012-04-19 06:53:02.935252] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:f9ce9cd7-92fa-4d32-b089-1ee7106178ee>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.935470] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.935513] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:02.936311] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:f9ce9cd7-92fa-4d32-b089-1ee7106178ee>.
>[2012-04-19 06:53:02.936332] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:f9ce9cd7-92fa-4d32-b089-1ee7106178ee>.
>[2012-04-19 06:53:02.936532] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:f9ce9cd7-92fa-4d32-b089-1ee7106178ee>
>[2012-04-19 06:53:02.936553] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:f9ce9cd7-92fa-4d32-b089-1ee7106178ee> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:02.937085] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:f9ce9cd7-92fa-4d32-b089-1ee7106178ee>
>[2012-04-19 06:53:02.946325] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC on hosdu-client-2
>[2012-04-19 06:53:02.946610] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/TIPS.DOC on hosdu-client-2
>[2012-04-19 06:53:02.948939] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:52ae4d44-46a0-4a92-8080-9758a246dae5>, reason: lookup detected pending operations
>[2012-04-19 06:53:02.961643] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/COURSES.VAL on hosdu-client-2
>[2012-04-19 06:53:02.965393] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/CHANGED.DB on hosdu-client-2
>[2012-04-19 06:53:02.965691] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:02.965945] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/ANSWER.DB on hosdu-client-2
>[2012-04-19 06:53:02.966502] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/HANDOUT.DOC on hosdu-client-2
>[2012-04-19 06:53:02.966685] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/TIPS.DOC on hosdu-client-2
>[2012-04-19 06:53:03.007604] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:52ae4d44-46a0-4a92-8080-9758a246dae5>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:03.010662] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:52ae4d44-46a0-4a92-8080-9758a246dae5>
>[2012-04-19 06:53:03.022168] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD, reason: lookup detected pending operations
>[2012-04-19 06:53:03.022319] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:e71ffa71-4af2-43d4-8d3f-3255136f9da8>.
>[2012-04-19 06:53:03.022340] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:e71ffa71-4af2-43d4-8d3f-3255136f9da8>, reason: lookup detected pending operations
>[2012-04-19 06:53:03.023014] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:03.093594] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>/COURSES.VAL on hosdu-client-2
>[2012-04-19 06:53:03.103523] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>/CHANGED.DB on hosdu-client-2
>[2012-04-19 06:53:03.103817] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:03.104078] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>/ANSWER.DB on hosdu-client-2
>[2012-04-19 06:53:03.117701] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWTIPS.PPT on hosdu-client-2
>[2012-04-19 06:53:03.118763] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:a22d5213-e34b-40cb-b757-12f096e97680>/NEWPCB.PPT on hosdu-client-2
>[2012-04-19 06:53:03.141034] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/COURSES.VAL on hosdu-client-2
>[2012-04-19 06:53:03.141288] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/CHANGED.DB on hosdu-client-2
>[2012-04-19 06:53:03.141566] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:03.141827] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/ANSWER.DB on hosdu-client-2
>[2012-04-19 06:53:03.151159] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:b610ee13-380b-4382-bf67-364f0cf8af2b>, reason: lookup detected pending operations
>[2012-04-19 06:53:03.160128] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>/COURSES.VAL on hosdu-client-2
>[2012-04-19 06:53:03.160389] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>/CHANGED.DB on hosdu-client-2
>[2012-04-19 06:53:03.160638] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:03.160882] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>/ANSWER.DB on hosdu-client-2
>[2012-04-19 06:53:03.171397] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWTIPS.PPT on hosdu-client-2
>[2012-04-19 06:53:03.177953] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWTIPS.PPT on hosdu-client-2
>[2012-04-19 06:53:03.752639] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:e71ffa71-4af2-43d4-8d3f-3255136f9da8>.
>[2012-04-19 06:53:03.752711] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:e71ffa71-4af2-43d4-8d3f-3255136f9da8>.
>[2012-04-19 06:53:03.868625] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:03.868840] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/~WRD0003.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:03.870042] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>/HANDOUT.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:03.899990] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/COURSES.VAL on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:03.906947] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:03.917256] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/HANDOUT.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:03.917475] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/TIPS.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:03.918262] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:e71ffa71-4af2-43d4-8d3f-3255136f9da8>
>[2012-04-19 06:53:03.918292] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:e71ffa71-4af2-43d4-8d3f-3255136f9da8> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:03.918314] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:e71ffa71-4af2-43d4-8d3f-3255136f9da8>
>[2012-04-19 06:53:03.925108] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:03.977685] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:03.980441] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:b610ee13-380b-4382-bf67-364f0cf8af2b>, reason: lookup detected pending operations
>[2012-04-19 06:53:04.797977] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>/~WRL0004.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:04.998106] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/HANDOUT.DOC on hosdu-client-2
>[2012-04-19 06:53:05.017111] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/TIPS.DOC on hosdu-client-2
>[2012-04-19 06:53:05.705614] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>/__S31.DB on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.706637] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWTIPS.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.721931] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:a22d5213-e34b-40cb-b757-12f096e97680>/NEWPCB.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.779147] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>/COURSES.VAL on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.779298] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>/CHANGED.DB on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.790288] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWTIPS.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.799233] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWTIPS.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.799478] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/PPTC241.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:05.956192] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:bc601fe2-25c7-4308-af07-fd0a5063180b>/SEED/SMALL.FIL: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:05.973747] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:624c2f0f-92fe-4de5-a71f-def2b4b4bde7>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:06.248509] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~$CHAP10.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:06.743671] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>
>[2012-04-19 06:53:06.743927] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:25c14e39-4fd2-47cb-9b22-59e97e49f92c>
>[2012-04-19 06:53:06.744072] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>
>[2012-04-19 06:53:06.744306] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>
>[2012-04-19 06:53:06.744557] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>
>[2012-04-19 06:53:06.745657] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC, reason: lookup detected pending operations
>[2012-04-19 06:53:06.757661] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP, reason: lookup detected pending operations
>[2012-04-19 06:53:06.757880] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/COURSES.DB, reason: lookup detected pending operations
>[2012-04-19 06:53:06.763861] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/COURSES.DB: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:06.766464] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:c200bd49-a1f7-4169-a2bc-8c3270e2a9b8>/COURSES.DB
>[2012-04-19 06:53:06.791401] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:1c579f03-638c-46c9-8f29-3172e5acb9d0>
>[2012-04-19 06:53:06.792621] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:9088f80d-7225-4e65-8e85-dd0341850f89>
>[2012-04-19 06:53:06.793172] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:caeacc73-2956-4210-8166-3ed512edb549>
>[2012-04-19 06:53:06.793363] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:a22d5213-e34b-40cb-b757-12f096e97680>
>[2012-04-19 06:53:06.795325] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:a22d5213-e34b-40cb-b757-12f096e97680>/NEWTIPS.PPT, reason: lookup detected pending operations
>[2012-04-19 06:53:06.796293] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:83b46a2e-0628-4a36-a7d2-f9f6e135bf18>
>[2012-04-19 06:53:06.796440] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT
>[2012-04-19 06:53:06.811433] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWTIPS.PPT 
>[2012-04-19 06:53:06.811470] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWTIPS.PPT, reason: lookup detected pending operations
>[2012-04-19 06:53:06.812710] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:a22d5213-e34b-40cb-b757-12f096e97680>/NEWTIPS.PPT
>[2012-04-19 06:53:06.815405] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:a22d5213-e34b-40cb-b757-12f096e97680>/NEWTIPS.PPT
>[2012-04-19 06:53:06.830039] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:3222e593-26cb-456c-b708-463c6528528f>, reason: lookup detected pending operations
>[2012-04-19 06:53:06.830242] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:2067838a-2374-459a-8bf7-79b66d79d34f> 
>[2012-04-19 06:53:06.830284] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:2067838a-2374-459a-8bf7-79b66d79d34f>, reason: lookup detected pending operations
>[2012-04-19 06:53:06.849051] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:af15a39b-51a8-4466-9c62-bef8beb3e644>, reason: lookup detected pending operations
>[2012-04-19 06:53:06.934621] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/COURSES.DB, reason: lookup detected pending operations
>[2012-04-19 06:53:06.943210] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:3222e593-26cb-456c-b708-463c6528528f>, reason: lookup detected pending operations
>[2012-04-19 06:53:06.943843] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:3222e593-26cb-456c-b708-463c6528528f>, reason: lookup detected pending operations
>[2012-04-19 06:53:06.948674] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/COURSES.DB: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:06.953168] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:7b92d3c9-b0f4-4d72-ad76-d7b0f8b06d54>/COURSES.DB
>[2012-04-19 06:53:06.966097] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:af15a39b-51a8-4466-9c62-bef8beb3e644>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:06.968336] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:af15a39b-51a8-4466-9c62-bef8beb3e644>
>[2012-04-19 06:53:06.986123] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:3222e593-26cb-456c-b708-463c6528528f>, reason: lookup detected pending operations
>[2012-04-19 06:53:06.996496] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:06.997627] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:07.022954] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:3222e593-26cb-456c-b708-463c6528528f>, reason: lookup detected pending operations
>[2012-04-19 06:53:07.023042] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:3222e593-26cb-456c-b708-463c6528528f>, reason: lookup detected pending operations
>[2012-04-19 06:53:07.511350] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>
>[2012-04-19 06:53:07.557059] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:20d942fe-fe7f-4317-8a2d-d4be67a950bc>, reason: lookup detected pending operations
>[2012-04-19 06:53:07.997651] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:bc601fe2-25c7-4308-af07-fd0a5063180b>/SEED/SMALL.FIL
>[2012-04-19 06:53:07.998363] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:624c2f0f-92fe-4de5-a71f-def2b4b4bde7>
>[2012-04-19 06:53:08.339592] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD
>[2012-04-19 06:53:08.341226] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD, reason: lookup detected pending operations
>[2012-04-19 06:53:08.345580] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~WRD0000.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:08.349206] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD
>[2012-04-19 06:53:08.352720] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~WRD0000.TMP 
>[2012-04-19 06:53:08.352747] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~WRD0000.TMP, reason: lookup detected pending operations
>[2012-04-19 06:53:08.359322] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~WRD0000.TMP
>[2012-04-19 06:53:08.362756] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~WRD0000.TMP
>[2012-04-19 06:53:08.542940] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC
>[2012-04-19 06:53:08.543228] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC 
>[2012-04-19 06:53:08.543251] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC, reason: lookup detected pending operations
>[2012-04-19 06:53:08.550529] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP 
>[2012-04-19 06:53:08.550576] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP, reason: lookup detected pending operations
>[2012-04-19 06:53:08.550685] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP
>[2012-04-19 06:53:08.560224] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:b610ee13-380b-4382-bf67-364f0cf8af2b>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:08.561873] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:20d942fe-fe7f-4317-8a2d-d4be67a950bc>
>[2012-04-19 06:53:08.567834] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWTIPS.PPT
>[2012-04-19 06:53:08.568998] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:2067838a-2374-459a-8bf7-79b66d79d34f>
>[2012-04-19 06:53:08.569188] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC
>[2012-04-19 06:53:08.571864] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP
>[2012-04-19 06:53:08.576552] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:b610ee13-380b-4382-bf67-364f0cf8af2b>
>[2012-04-19 06:53:08.578583] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:20d942fe-fe7f-4317-8a2d-d4be67a950bc>
>[2012-04-19 06:53:08.579948] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:caeacc73-2956-4210-8166-3ed512edb549>/NEWTIPS.PPT
>[2012-04-19 06:53:08.581939] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:b610ee13-380b-4382-bf67-364f0cf8af2b>
>[2012-04-19 06:53:08.582624] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:2067838a-2374-459a-8bf7-79b66d79d34f>
>[2012-04-19 06:53:08.589922] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:f87dea22-e793-44fb-8f11-2dda2bfcbd04>.
>[2012-04-19 06:53:08.589953] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:f87dea22-e793-44fb-8f11-2dda2bfcbd04>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.592006] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.592388] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.595287] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:407409c0-8828-4caa-985d-7cd711db290d>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.595588] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.596400] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:f87dea22-e793-44fb-8f11-2dda2bfcbd04>.
>[2012-04-19 06:53:08.596422] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:f87dea22-e793-44fb-8f11-2dda2bfcbd04>.
>[2012-04-19 06:53:08.596715] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:407409c0-8828-4caa-985d-7cd711db290d>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.598935] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:f87dea22-e793-44fb-8f11-2dda2bfcbd04>
>[2012-04-19 06:53:08.598958] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:f87dea22-e793-44fb-8f11-2dda2bfcbd04> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:08.600021] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:f87dea22-e793-44fb-8f11-2dda2bfcbd04>
>[2012-04-19 06:53:08.600861] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.603976] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWTIPS.PPT 
>[2012-04-19 06:53:08.604033] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWTIPS.PPT, reason: lookup detected pending operations
>[2012-04-19 06:53:08.605105] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:407409c0-8828-4caa-985d-7cd711db290d>/HANDOUT.DOC on hosdu-client-1
>[2012-04-19 06:53:08.605662] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:407409c0-8828-4caa-985d-7cd711db290d>/TIPS.DOC on hosdu-client-1
>[2012-04-19 06:53:08.608715] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.610688] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.611960] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:e872f4f5-81fc-4f3f-a850-a4aa02ef1eb4>.
>[2012-04-19 06:53:08.611984] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:e872f4f5-81fc-4f3f-a850-a4aa02ef1eb4>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.613221] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.616335] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/~WRL0004.TMP 
>[2012-04-19 06:53:08.616985] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:08.617301] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:407409c0-8828-4caa-985d-7cd711db290d>/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:08.617484] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRD1119.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:08.618302] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC
>[2012-04-19 06:53:08.619205] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP
>[2012-04-19 06:53:08.619578] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:e872f4f5-81fc-4f3f-a850-a4aa02ef1eb4>.
>[2012-04-19 06:53:08.619598] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:e872f4f5-81fc-4f3f-a850-a4aa02ef1eb4>.
>[2012-04-19 06:53:08.620924] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRL0004.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:08.621045] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRL1146.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:08.635177] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:e872f4f5-81fc-4f3f-a850-a4aa02ef1eb4>
>[2012-04-19 06:53:08.635205] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:e872f4f5-81fc-4f3f-a850-a4aa02ef1eb4> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:08.635689] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:e872f4f5-81fc-4f3f-a850-a4aa02ef1eb4>
>[2012-04-19 06:53:08.646356] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.651140] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.665322] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:3222e593-26cb-456c-b708-463c6528528f>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:08.666453] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:407409c0-8828-4caa-985d-7cd711db290d>
>[2012-04-19 06:53:08.695175] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWTIPS.PPT
>[2012-04-19 06:53:08.725100] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:3222e593-26cb-456c-b708-463c6528528f>
>[2012-04-19 06:53:08.725980] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:609409cf-c223-47cd-88c1-829da36058f1>.
>[2012-04-19 06:53:08.726006] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:609409cf-c223-47cd-88c1-829da36058f1>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.733199] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.741266] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:3222e593-26cb-456c-b708-463c6528528f>
>[2012-04-19 06:53:08.749871] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:94de7089-bbcb-4d68-b01a-b3e11d36e30c>/NEWTIPS.PPT
>[2012-04-19 06:53:08.757177] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX, reason: lookup detected pending operations
>[2012-04-19 06:53:08.758541] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:3222e593-26cb-456c-b708-463c6528528f>
>[2012-04-19 06:53:08.798015] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:a951aab1-9c31-4e8e-8be5-c5655289832d>.
>[2012-04-19 06:53:08.798061] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:a951aab1-9c31-4e8e-8be5-c5655289832d>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.798632] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:08.848966] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:1694f8bd-d4ba-4c46-8bb8-3675273f3448>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.937991] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:16436edb-ef2c-4857-8108-aec9e2fb0ff3>.
>[2012-04-19 06:53:08.938029] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:16436edb-ef2c-4857-8108-aec9e2fb0ff3>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.978655] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:179080b8-951b-400c-86ae-2a4ed409262e>.
>[2012-04-19 06:53:08.978692] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:179080b8-951b-400c-86ae-2a4ed409262e>, reason: lookup detected pending operations
>[2012-04-19 06:53:08.980933] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>, reason: lookup detected pending operations
>[2012-04-19 06:53:09.004393] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>, reason: lookup detected pending operations
>[2012-04-19 06:53:09.005001] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD, reason: lookup detected pending operations
>[2012-04-19 06:53:09.117012] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.117383] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:609409cf-c223-47cd-88c1-829da36058f1>.
>[2012-04-19 06:53:09.117404] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:609409cf-c223-47cd-88c1-829da36058f1>.
>[2012-04-19 06:53:09.122513] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:3222e593-26cb-456c-b708-463c6528528f>
>[2012-04-19 06:53:09.284167] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.284545] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:a951aab1-9c31-4e8e-8be5-c5655289832d>.
>[2012-04-19 06:53:09.284566] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:a951aab1-9c31-4e8e-8be5-c5655289832d>.
>[2012-04-19 06:53:09.624292] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.624721] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:16436edb-ef2c-4857-8108-aec9e2fb0ff3>.
>[2012-04-19 06:53:09.624741] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:16436edb-ef2c-4857-8108-aec9e2fb0ff3>.
>[2012-04-19 06:53:09.660859] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.661279] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:179080b8-951b-400c-86ae-2a4ed409262e>.
>[2012-04-19 06:53:09.661299] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:179080b8-951b-400c-86ae-2a4ed409262e>.
>[2012-04-19 06:53:09.729040] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:609409cf-c223-47cd-88c1-829da36058f1>
>[2012-04-19 06:53:09.729071] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:609409cf-c223-47cd-88c1-829da36058f1> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:09.729087] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:609409cf-c223-47cd-88c1-829da36058f1>
>[2012-04-19 06:53:09.739876] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.740368] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.740899] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.742022] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.742649] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.761385] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWTIPS.PPT 
>[2012-04-19 06:53:09.761410] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWTIPS.PPT, reason: lookup detected pending operations
>[2012-04-19 06:53:09.870172] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:a951aab1-9c31-4e8e-8be5-c5655289832d>
>[2012-04-19 06:53:09.870208] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:a951aab1-9c31-4e8e-8be5-c5655289832d> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:09.870226] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:a951aab1-9c31-4e8e-8be5-c5655289832d>
>[2012-04-19 06:53:09.870578] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.871405] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.871982] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:09.873005] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:9d5453fb-82bd-40ed-992d-3c8ae2833756>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.179546] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:16436edb-ef2c-4857-8108-aec9e2fb0ff3>
>[2012-04-19 06:53:10.179591] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:16436edb-ef2c-4857-8108-aec9e2fb0ff3> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:10.179607] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:16436edb-ef2c-4857-8108-aec9e2fb0ff3>
>[2012-04-19 06:53:10.180019] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.180983] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.181226] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.181838] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.182564] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.182880] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.184684] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.217421] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:179080b8-951b-400c-86ae-2a4ed409262e>
>[2012-04-19 06:53:10.217445] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:179080b8-951b-400c-86ae-2a4ed409262e> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:10.217459] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:179080b8-951b-400c-86ae-2a4ed409262e>
>[2012-04-19 06:53:10.217878] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.218733] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.219056] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.220307] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:d6179b01-4979-4913-a1e7-75bcfa954669>.
>[2012-04-19 06:53:10.220331] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:d6179b01-4979-4913-a1e7-75bcfa954669>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.325810] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:407409c0-8828-4caa-985d-7cd711db290d>
>[2012-04-19 06:53:10.330825] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRD1119.TMP 
>[2012-04-19 06:53:10.330851] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRD1119.TMP, reason: lookup detected pending operations
>[2012-04-19 06:53:10.338449] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.339083] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.339341] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.339946] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:d27ffeba-9b3a-4c60-ae0c-698d0e152803>.
>[2012-04-19 06:53:10.339969] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:d27ffeba-9b3a-4c60-ae0c-698d0e152803>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.344799] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/COURSES.VAL on hosdu-client-1
>[2012-04-19 06:53:10.344859] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/CHANGED.DB on hosdu-client-1
>[2012-04-19 06:53:10.356702] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:10.357138] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/ANSWER.DB on hosdu-client-1
>[2012-04-19 06:53:10.407513] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.407829] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.407890] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:d6179b01-4979-4913-a1e7-75bcfa954669>.
>[2012-04-19 06:53:10.407906] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:d6179b01-4979-4913-a1e7-75bcfa954669>.
>[2012-04-19 06:53:10.407964] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:3222e593-26cb-456c-b708-463c6528528f>
>[2012-04-19 06:53:10.408433] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:d27ffeba-9b3a-4c60-ae0c-698d0e152803>.
>[2012-04-19 06:53:10.408454] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:d27ffeba-9b3a-4c60-ae0c-698d0e152803>.
>[2012-04-19 06:53:10.439853] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:598bc6e2-a8f0-44f5-adbf-9132ebbfe082>/~WRL0004.TMP
>[2012-04-19 06:53:10.506993] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:d6179b01-4979-4913-a1e7-75bcfa954669>
>[2012-04-19 06:53:10.507023] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:d6179b01-4979-4913-a1e7-75bcfa954669> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:10.507038] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:d6179b01-4979-4913-a1e7-75bcfa954669>
>[2012-04-19 06:53:10.507321] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:d27ffeba-9b3a-4c60-ae0c-698d0e152803>
>[2012-04-19 06:53:10.507342] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:d27ffeba-9b3a-4c60-ae0c-698d0e152803> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:10.507355] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:d27ffeba-9b3a-4c60-ae0c-698d0e152803>
>[2012-04-19 06:53:10.509143] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>/COURSES.VAL on hosdu-client-2
>[2012-04-19 06:53:10.510237] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.510546] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.523702] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>/CHANGED.DB on hosdu-client-2
>[2012-04-19 06:53:10.524046] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:10.524401] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>/ANSWER.DB on hosdu-client-2
>[2012-04-19 06:53:10.525153] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.525252] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.526074] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.548495] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.560760] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:10.561019] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:3222e593-26cb-456c-b708-463c6528528f>
>[2012-04-19 06:53:10.577983] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:ef359815-f506-4d45-af0e-b1d08633851d>.
>[2012-04-19 06:53:10.578054] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:ef359815-f506-4d45-af0e-b1d08633851d>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.578293] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.592893] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.602944] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:7fdfdb6b-d5ea-4c67-b9cb-64797c7931c7>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.603273] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:ee028816-313d-40ab-9a8c-743b4aabf379>.
>[2012-04-19 06:53:10.603295] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:ee028816-313d-40ab-9a8c-743b4aabf379>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.603444] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.608445] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.618082] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.618937] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.625805] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:539cbc23-15c3-4f55-b548-b6bc62a6a454>.
>[2012-04-19 06:53:10.625828] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:539cbc23-15c3-4f55-b548-b6bc62a6a454>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.627830] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.627885] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:d2902048-fa24-4fda-99ed-729a4038de7a>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.629798] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.632148] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:d2902048-fa24-4fda-99ed-729a4038de7a> 
>[2012-04-19 06:53:10.632174] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:d2902048-fa24-4fda-99ed-729a4038de7a>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.666445] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.668045] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.672833] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:b3cfc5d1-b998-4195-81d4-559fe4b509a3> 
>[2012-04-19 06:53:10.672857] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:b3cfc5d1-b998-4195-81d4-559fe4b509a3>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.683582] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.685848] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.713434] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.719851] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:10.792439] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:cb423488-9026-4c0b-b5f5-c517b8873889>.
>[2012-04-19 06:53:10.792469] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:cb423488-9026-4c0b-b5f5-c517b8873889>, reason: lookup detected pending operations
>[2012-04-19 06:53:10.792828] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:10.800434] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.038308] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:ee028816-313d-40ab-9a8c-743b4aabf379>.
>[2012-04-19 06:53:11.038343] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:ee028816-313d-40ab-9a8c-743b4aabf379>.
>[2012-04-19 06:53:11.086632] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:19842d4e-a3b3-494c-8b54-75cb9f3660bc> 
>[2012-04-19 06:53:11.086669] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:19842d4e-a3b3-494c-8b54-75cb9f3660bc>, reason: lookup detected pending operations
>[2012-04-19 06:53:11.102256] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:a1758292-3f06-4313-91cd-6072d66ad7ce>/HANDOUT.DOC
>[2012-04-19 06:53:11.183212] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.183555] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:cb423488-9026-4c0b-b5f5-c517b8873889>.
>[2012-04-19 06:53:11.183575] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:cb423488-9026-4c0b-b5f5-c517b8873889>.
>[2012-04-19 06:53:11.253076] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:ef359815-f506-4d45-af0e-b1d08633851d>.
>[2012-04-19 06:53:11.253102] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:ef359815-f506-4d45-af0e-b1d08633851d>.
>[2012-04-19 06:53:11.253309] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:539cbc23-15c3-4f55-b548-b6bc62a6a454>.
>[2012-04-19 06:53:11.253329] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:539cbc23-15c3-4f55-b548-b6bc62a6a454>.
>[2012-04-19 06:53:11.303962] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:ee028816-313d-40ab-9a8c-743b4aabf379>
>[2012-04-19 06:53:11.303991] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:ee028816-313d-40ab-9a8c-743b4aabf379> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:11.304007] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:ee028816-313d-40ab-9a8c-743b4aabf379>
>[2012-04-19 06:53:11.305110] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.308836] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.349325] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.354133] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:f42c0f5d-0961-490f-871d-85410e486fff>.
>[2012-04-19 06:53:11.354158] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:f42c0f5d-0961-490f-871d-85410e486fff>, reason: lookup detected pending operations
>[2012-04-19 06:53:11.464293] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>/COURSES.VAL on hosdu-client-1
>[2012-04-19 06:53:11.466491] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>/CHANGED.DB on hosdu-client-1
>[2012-04-19 06:53:11.482616] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:11.482886] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>/ANSWER.DB on hosdu-client-1
>[2012-04-19 06:53:11.757311] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:11.757986] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/~WRL1146.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:11.758693] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:ef359815-f506-4d45-af0e-b1d08633851d>
>[2012-04-19 06:53:11.758715] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:ef359815-f506-4d45-af0e-b1d08633851d> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:11.758730] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:ef359815-f506-4d45-af0e-b1d08633851d>
>[2012-04-19 06:53:11.764905] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.765067] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:539cbc23-15c3-4f55-b548-b6bc62a6a454>
>[2012-04-19 06:53:11.765087] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:539cbc23-15c3-4f55-b548-b6bc62a6a454> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:11.765099] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:539cbc23-15c3-4f55-b548-b6bc62a6a454>
>[2012-04-19 06:53:11.765412] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.765684] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.766196] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.769259] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.769818] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.770924] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:4e645de3-3e12-4c62-886a-49f7be3c3ec2>.
>[2012-04-19 06:53:11.770947] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:4e645de3-3e12-4c62-886a-49f7be3c3ec2>, reason: lookup detected pending operations
>[2012-04-19 06:53:11.774840] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.779315] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:cb423488-9026-4c0b-b5f5-c517b8873889>
>[2012-04-19 06:53:11.779339] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:cb423488-9026-4c0b-b5f5-c517b8873889> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:11.779352] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:cb423488-9026-4c0b-b5f5-c517b8873889>
>[2012-04-19 06:53:11.779929] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.816406] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.816791] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:f42c0f5d-0961-490f-871d-85410e486fff>.
>[2012-04-19 06:53:11.816812] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:f42c0f5d-0961-490f-871d-85410e486fff>.
>[2012-04-19 06:53:11.821505] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:11.849486] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:3e1e2e46-741e-4b21-8c9c-d38231a46801>, reason: lookup detected pending operations
>[2012-04-19 06:53:11.850216] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>, reason: lookup detected pending operations
>[2012-04-19 06:53:11.889164] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>, reason: lookup detected pending operations
>[2012-04-19 06:53:12.124610] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>
>[2012-04-19 06:53:12.126828] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:f42c0f5d-0961-490f-871d-85410e486fff>
>[2012-04-19 06:53:12.126851] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:f42c0f5d-0961-490f-871d-85410e486fff> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:12.126866] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:f42c0f5d-0961-490f-871d-85410e486fff>
>[2012-04-19 06:53:12.127156] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:12.128164] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:12.157793] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>, reason: lookup detected pending operations
>[2012-04-19 06:53:12.159741] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>, reason: lookup detected pending operations
>[2012-04-19 06:53:12.163296] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:12.175199] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:12.175293] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:4e645de3-3e12-4c62-886a-49f7be3c3ec2>.
>[2012-04-19 06:53:12.175312] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:4e645de3-3e12-4c62-886a-49f7be3c3ec2>.
>[2012-04-19 06:53:12.285844] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX
>[2012-04-19 06:53:12.286710] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX, reason: lookup detected pending operations
>[2012-04-19 06:53:12.325727] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/COURSES.VAL on hosdu-client-1
>[2012-04-19 06:53:12.326504] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/CHANGED.DB on hosdu-client-1
>[2012-04-19 06:53:12.327151] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:12.339806] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/ANSWER.DB on hosdu-client-1
>[2012-04-19 06:53:12.340208] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:4e645de3-3e12-4c62-886a-49f7be3c3ec2>
>[2012-04-19 06:53:12.340230] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:4e645de3-3e12-4c62-886a-49f7be3c3ec2> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:12.340245] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:4e645de3-3e12-4c62-886a-49f7be3c3ec2>
>[2012-04-19 06:53:12.340794] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:12.343951] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>/HANDOUT.DOC on hosdu-client-1
>[2012-04-19 06:53:12.360516] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>/TIPS.DOC on hosdu-client-1
>[2012-04-19 06:53:12.383304] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:12.383704] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:12.388522] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>, reason: lookup detected pending operations
>[2012-04-19 06:53:12.402910] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>, reason: lookup detected pending operations
>[2012-04-19 06:53:12.419921] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:22cdcdc1-a2dc-4f74-a679-3c425196228a>
>[2012-04-19 06:53:12.445152] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>
>[2012-04-19 06:53:12.453924] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWTIPS.PPT
>[2012-04-19 06:53:12.459817] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>/~WRL0004.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:12.459854] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>/~WRL1146.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:12.468458] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWPCB.PPT on hosdu-client-2
>[2012-04-19 06:53:12.523131] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:12.547839] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>, reason: lookup detected pending operations
>[2012-04-19 06:53:12.562824] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>, reason: lookup detected pending operations
>[2012-04-19 06:53:12.957816] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:0fb6bd55-b89d-4136-89b2-df361addb9d2> 
>[2012-04-19 06:53:12.957862] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:0fb6bd55-b89d-4136-89b2-df361addb9d2>, reason: lookup detected pending operations
>[2012-04-19 06:53:13.003853] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:13.004439] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:13.006287] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:13.007019] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:13.008626] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:13.009170] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:13.009598] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:13.011845] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:db58d035-3bb9-4686-a905-476fd39cd9f5>, reason: lookup detected pending operations
>[2012-04-19 06:53:13.093934] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>/COURSES.VAL on hosdu-client-2
>[2012-04-19 06:53:13.094170] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>/CHANGED.DB on hosdu-client-2
>[2012-04-19 06:53:13.100420] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>/ANSWER.MB on hosdu-client-2
>[2012-04-19 06:53:13.100709] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>/ANSWER.DB on hosdu-client-2
>[2012-04-19 06:53:13.151480] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>/COURSES.VAL on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:13.151531] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>/CHANGED.DB on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:13.171575] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWTIPS.PPT on hosdu-client-1
>[2012-04-19 06:53:13.185044] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:9d5453fb-82bd-40ed-992d-3c8ae2833756>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:13.215608] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/HANDOUT.DOC on hosdu-client-1
>[2012-04-19 06:53:13.215861] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/TIPS.DOC on hosdu-client-1
>[2012-04-19 06:53:13.274954] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWPCB.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:13.290273] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/~WRD0000.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:13.480200] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/__S31.DB on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:13.480583] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/COURSES.VAL on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:13.499047] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:13.501075] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:13.503064] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>/TIPS.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:13.582066] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD
>[2012-04-19 06:53:13.582854] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD, reason: lookup detected pending operations
>[2012-04-19 06:53:13.624820] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD, reason: lookup detected pending operations
>[2012-04-19 06:53:13.724299] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>/ANSWER.MB on hosdu-client-1
>[2012-04-19 06:53:13.767564] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:d2902048-fa24-4fda-99ed-729a4038de7a>
>[2012-04-19 06:53:13.916908] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRD1119.TMP
>[2012-04-19 06:53:13.995414] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/PPTC241.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:14.001027] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWTIPS.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:14.140052] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>
>[2012-04-19 06:53:14.172855] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>
>[2012-04-19 06:53:14.213656] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:d2902048-fa24-4fda-99ed-729a4038de7a>
>[2012-04-19 06:53:14.213762] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:b3cfc5d1-b998-4195-81d4-559fe4b509a3>
>[2012-04-19 06:53:14.231170] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:14.258871] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.274251] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.274847] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.275799] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.276237] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.276722] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.289805] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX
>[2012-04-19 06:53:14.290816] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>/HANDOUT.DOC on hosdu-client-2
>[2012-04-19 06:53:14.290999] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>/TIPS.DOC on hosdu-client-2
>[2012-04-19 06:53:14.297090] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:19842d4e-a3b3-494c-8b54-75cb9f3660bc>
>[2012-04-19 06:53:14.297175] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:4391ad28-cfcd-42c9-99d4-f6f78f21f658>/PWRPNT/NEWTIPS.PPT
>[2012-04-19 06:53:14.298087] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/~$CHAP10.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:14.363318] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~$CHAP10.DOC 
>[2012-04-19 06:53:14.363342] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~$CHAP10.DOC, reason: lookup detected pending operations
>[2012-04-19 06:53:14.400993] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/HANDOUT.DOC on hosdu-client-2
>[2012-04-19 06:53:14.401188] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/TIPS.DOC on hosdu-client-2
>[2012-04-19 06:53:14.446190] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>
>[2012-04-19 06:53:14.446321] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>
>[2012-04-19 06:53:14.535797] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/COURSES.PX, reason: lookup detected pending operations
>[2012-04-19 06:53:14.622484] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:a76592e6-7773-41ff-868d-9f5c7324483a>
>[2012-04-19 06:53:14.622765] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD/~WRL0004.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:14.622946] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD/~$FACTS.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:14.623269] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>
>[2012-04-19 06:53:14.624926] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:d2902048-fa24-4fda-99ed-729a4038de7a>
>[2012-04-19 06:53:14.749167] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.749608] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.751684] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>, reason: lookup detected pending operations
>[2012-04-19 06:53:14.752009] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>, reason: lookup detected pending operations
>[2012-04-19 06:53:14.807894] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>
>[2012-04-19 06:53:14.808140] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:c94a53bb-a9e8-4052-822d-a2575757e3f8>
>[2012-04-19 06:53:14.809478] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/__S31.DB 
>[2012-04-19 06:53:14.809501] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/__S31.DB, reason: lookup detected pending operations
>[2012-04-19 06:53:14.810568] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:0f77acc9-ecb0-4f7d-9cb5-9f0817457d17>.
>[2012-04-19 06:53:14.810591] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:0f77acc9-ecb0-4f7d-9cb5-9f0817457d17>, reason: lookup detected pending operations
>[2012-04-19 06:53:14.820869] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:9d5453fb-82bd-40ed-992d-3c8ae2833756>
>[2012-04-19 06:53:14.860457] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.861088] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:0f77acc9-ecb0-4f7d-9cb5-9f0817457d17>.
>[2012-04-19 06:53:14.861112] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:0f77acc9-ecb0-4f7d-9cb5-9f0817457d17>.
>[2012-04-19 06:53:14.873302] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:1694f8bd-d4ba-4c46-8bb8-3675273f3448>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:14.874852] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:d2902048-fa24-4fda-99ed-729a4038de7a>
>[2012-04-19 06:53:14.876128] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>.
>[2012-04-19 06:53:14.876152] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>, reason: lookup detected pending operations
>[2012-04-19 06:53:14.876960] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>.
>[2012-04-19 06:53:14.876983] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>, reason: lookup detected pending operations
>[2012-04-19 06:53:14.878095] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:3b4dcb14-a5c0-4bd1-ae5c-14bb163d48af>
>[2012-04-19 06:53:14.880850] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:3f85ffee-3514-4cd9-b1a0-9568c102fbd6>, reason: lookup detected pending operations
>[2012-04-19 06:53:14.881087] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:0f77acc9-ecb0-4f7d-9cb5-9f0817457d17>
>[2012-04-19 06:53:14.881109] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:0f77acc9-ecb0-4f7d-9cb5-9f0817457d17> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:14.881135] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:0f77acc9-ecb0-4f7d-9cb5-9f0817457d17>
>[2012-04-19 06:53:14.898552] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:14.898834] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:0e96076f-ef17-4054-a346-c348907fe704>, reason: lookup detected pending operations
>[2012-04-19 06:53:14.898963] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>.
>[2012-04-19 06:53:14.898982] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>.
>[2012-04-19 06:53:14.953265] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT, reason: lookup detected pending operations
>[2012-04-19 06:53:15.021504] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.021752] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.024158] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.024319] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.024474] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>
>[2012-04-19 06:53:15.024501] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:15.024522] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>
>[2012-04-19 06:53:15.026351] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:dab2e806-58e2-45bd-bd7f-3e1130ff2d7b>.
>[2012-04-19 06:53:15.026375] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:dab2e806-58e2-45bd-bd7f-3e1130ff2d7b>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.032811] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/NEWTIPS.PPT on hosdu-client-1
>[2012-04-19 06:53:15.069375] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.069718] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.070218] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.072076] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.072273] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.072573] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.075814] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:407409c0-8828-4caa-985d-7cd711db290d>/~WRD1119.TMP
>[2012-04-19 06:53:15.076194] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.077876] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:4c66f717-1fa9-4459-ae6c-175e707fd3ec>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.079803] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.081375] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>
>[2012-04-19 06:53:15.081704] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>
>[2012-04-19 06:53:15.093080] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:0fb6bd55-b89d-4136-89b2-df361addb9d2>
>[2012-04-19 06:53:15.093368] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWTIPS.PPT 
>[2012-04-19 06:53:15.093390] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWTIPS.PPT, reason: lookup detected pending operations
>[2012-04-19 06:53:15.098820] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.099968] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.135164] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>
>[2012-04-19 06:53:15.136200] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/~WRD0000.TMP 
>[2012-04-19 06:53:15.136223] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/~WRD0000.TMP, reason: lookup detected pending operations
>[2012-04-19 06:53:15.136452] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:a932498a-5c74-4780-8169-88bf3bb17806>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.338543] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:b3cfc5d1-b998-4195-81d4-559fe4b509a3>
>[2012-04-19 06:53:15.344843] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:845c7a84-9eb1-4e44-84e1-27d5f257ad65> 
>[2012-04-19 06:53:15.344869] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:845c7a84-9eb1-4e44-84e1-27d5f257ad65>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.344957] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:845c7a84-9eb1-4e44-84e1-27d5f257ad65>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.356545] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>/HANDOUT.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:15.356912] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>/~WRD0003.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:15.552195] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:7edff096-5541-43c5-8b08-11a061847d81>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.584350] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.584835] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>.
>[2012-04-19 06:53:15.584857] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>.
>[2012-04-19 06:53:15.686342] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:db58d035-3bb9-4686-a905-476fd39cd9f5>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:15.730334] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:15.730841] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:dab2e806-58e2-45bd-bd7f-3e1130ff2d7b>.
>[2012-04-19 06:53:15.730862] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:dab2e806-58e2-45bd-bd7f-3e1130ff2d7b>.
>[2012-04-19 06:53:15.744003] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:352d0e0d-3ebe-49ea-90c1-47bc1ae4a9ae>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.831906] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/COURSES.PX: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:15.833072] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:1694f8bd-d4ba-4c46-8bb8-3675273f3448>
>[2012-04-19 06:53:15.877825] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:12ac305e-7cbd-4df7-804b-679d36645593>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.927625] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:3e1e2e46-741e-4b21-8c9c-d38231a46801>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:15.946245] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:b11644e5-3351-4842-a512-7135c71e2894>, reason: lookup detected pending operations
>[2012-04-19 06:53:15.956814] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/PPTC241.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:15.956881] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/NEWTIPS.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:15.976991] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:19842d4e-a3b3-494c-8b54-75cb9f3660bc>
>[2012-04-19 06:53:15.993245] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/~WRL0004.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:15.993510] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/HANDOUT.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:15.993807] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/TIPS.DOC on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:16.008889] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:f36dfc41-8fa4-4d69-995e-fd68e3c2e17f>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.101021] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>
>[2012-04-19 06:53:16.101046] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:16.101060] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:20d8f351-2e41-493b-bde4-f0cee1d8e001>
>[2012-04-19 06:53:16.167280] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:dab2e806-58e2-45bd-bd7f-3e1130ff2d7b>
>[2012-04-19 06:53:16.167305] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:dab2e806-58e2-45bd-bd7f-3e1130ff2d7b> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:16.167318] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:dab2e806-58e2-45bd-bd7f-3e1130ff2d7b>
>[2012-04-19 06:53:16.170392] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:9207e187-22e8-478d-a835-c496c692683e>.
>[2012-04-19 06:53:16.170417] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:9207e187-22e8-478d-a835-c496c692683e>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.170518] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:9207e187-22e8-478d-a835-c496c692683e>.
>[2012-04-19 06:53:16.170537] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:9207e187-22e8-478d-a835-c496c692683e>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.272152] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:16.272612] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:9207e187-22e8-478d-a835-c496c692683e>.
>[2012-04-19 06:53:16.272642] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:9207e187-22e8-478d-a835-c496c692683e>.
>[2012-04-19 06:53:16.273691] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:8c680689-8eac-40da-9724-ed739bbee102>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.382869] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/NEWPCB.PPT on hosdu-client-2
>[2012-04-19 06:53:16.384000] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/NEWTIPS.PPT on hosdu-client-2
>[2012-04-19 06:53:16.393949] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:0fb6bd55-b89d-4136-89b2-df361addb9d2>
>[2012-04-19 06:53:16.395838] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:9207e187-22e8-478d-a835-c496c692683e>
>[2012-04-19 06:53:16.395862] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:9207e187-22e8-478d-a835-c496c692683e> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:16.395876] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:9207e187-22e8-478d-a835-c496c692683e>
>[2012-04-19 06:53:16.396204] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD
>[2012-04-19 06:53:16.402898] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:0a0c637d-0a87-4825-82bd-9c01c87d993d>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.408467] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:3c31437c-4f7d-4356-ac31-111811fb1e3a>
>[2012-04-19 06:53:16.420587] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD
>[2012-04-19 06:53:16.435978] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:1401e39f-8e59-43e5-813a-277728bcdefd>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.436433] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:16.438647] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/NEWPCB.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:16.440196] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/NEWTIPS.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:16.441807] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/PPTC241.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:16.441852] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:16.441914] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~$CHAP10.DOC
>[2012-04-19 06:53:16.442129] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:16.445820] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD, reason: lookup detected pending operations
>[2012-04-19 06:53:16.457603] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/TIPS.DOC 
>[2012-04-19 06:53:16.457628] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/TIPS.DOC, reason: lookup detected pending operations
>[2012-04-19 06:53:16.473201] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:16.473681] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:9207e187-22e8-478d-a835-c496c692683e>.
>[2012-04-19 06:53:16.473708] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:9207e187-22e8-478d-a835-c496c692683e>.
>[2012-04-19 06:53:16.500452] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  data self-heal triggered. path: <gfid:5c02d1bd-5fad-4388-ab67-9e56de3a324a>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.612693] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.626522] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.628819] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/__S31.DB
>[2012-04-19 06:53:16.691579] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:9207e187-22e8-478d-a835-c496c692683e>
>[2012-04-19 06:53:16.691603] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:9207e187-22e8-478d-a835-c496c692683e> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:16.691622] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:9207e187-22e8-478d-a835-c496c692683e>
>[2012-04-19 06:53:16.708068] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:1f23432c-24e2-4867-8965-38ab529e2cdd>.
>[2012-04-19 06:53:16.708118] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:1f23432c-24e2-4867-8965-38ab529e2cdd>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.719063] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:3f85ffee-3514-4cd9-b1a0-9568c102fbd6>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:16.735171] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:db58d035-3bb9-4686-a905-476fd39cd9f5>
>[2012-04-19 06:53:16.758888] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:2e8e680e-e409-48c1-88e4-5bac8637cb3c>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.762331] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:497c0414-bddc-482b-a1ba-d579501c7586>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.769693] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:0e96076f-ef17-4054-a346-c348907fe704>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:16.774859] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:d58476d6-e07a-479d-b911-0188d317e4ca>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.827068] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:fb73c2aa-af5e-4348-bec4-eb28af0eba75>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.827371] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:d8dbc312-f52f-49a0-b85d-7169e4959217>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.875196] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>
>[2012-04-19 06:53:16.875651] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:4cfb7883-2a1d-42a2-8d7b-9feef4c698c2>/PARADOX/COURSES.PX
>[2012-04-19 06:53:16.886311] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:845c7a84-9eb1-4e44-84e1-27d5f257ad65>
>[2012-04-19 06:53:16.917830] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:ee757995-b923-47b1-9c4f-c84527c56c26>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.935648] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:ee757995-b923-47b1-9c4f-c84527c56c26> 
>[2012-04-19 06:53:16.935674] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:ee757995-b923-47b1-9c4f-c84527c56c26>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.940660] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:3e1e2e46-741e-4b21-8c9c-d38231a46801>
>[2012-04-19 06:53:16.956630] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD/HANDOUT.DOC on hosdu-client-1
>[2012-04-19 06:53:16.958805] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD/TIPS.DOC on hosdu-client-1
>[2012-04-19 06:53:16.968840] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>.
>[2012-04-19 06:53:16.968865] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>, reason: lookup detected pending operations
>[2012-04-19 06:53:16.968949] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>.
>[2012-04-19 06:53:16.968970] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>, reason: lookup detected pending operations
>[2012-04-19 06:53:17.007466] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/HANDOUT.DOC on hosdu-client-1
>[2012-04-19 06:53:17.007714] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/TIPS.DOC on hosdu-client-1
>[2012-04-19 06:53:17.098179] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:17.098650] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:1f23432c-24e2-4867-8965-38ab529e2cdd>.
>[2012-04-19 06:53:17.098671] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:1f23432c-24e2-4867-8965-38ab529e2cdd>.
>[2012-04-19 06:53:17.240652] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWTIPS.PPT
>[2012-04-19 06:53:17.274346] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:17.274841] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>.
>[2012-04-19 06:53:17.274867] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>.
>[2012-04-19 06:53:17.532823] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>
>[2012-04-19 06:53:17.532853] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:1684b9f9-7891-441a-b0cd-a871152a9439> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:17.532872] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>
>[2012-04-19 06:53:17.610423] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:17.610863] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>.
>[2012-04-19 06:53:17.610885] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>.
>[2012-04-19 06:53:17.659743] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:9315a1e4-fcfa-49a1-9f34-58689339e8c6>/__S31.DB
>[2012-04-19 06:53:17.666849] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:12ac305e-7cbd-4df7-804b-679d36645593>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:17.685156] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:a932498a-5c74-4780-8169-88bf3bb17806>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:17.702376] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:1f23432c-24e2-4867-8965-38ab529e2cdd>
>[2012-04-19 06:53:17.702405] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:1f23432c-24e2-4867-8965-38ab529e2cdd> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:17.702420] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:1f23432c-24e2-4867-8965-38ab529e2cdd>
>[2012-04-19 06:53:17.705749] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:511925c2-d287-48b4-9998-3d889326fb92> 
>[2012-04-19 06:53:17.705793] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:511925c2-d287-48b4-9998-3d889326fb92>, reason: lookup detected pending operations
>[2012-04-19 06:53:17.762237] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:3f85ffee-3514-4cd9-b1a0-9568c102fbd6>
>[2012-04-19 06:53:17.787898] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:b11644e5-3351-4842-a512-7135c71e2894>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:17.830404] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:1401e39f-8e59-43e5-813a-277728bcdefd>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:17.838854] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:0e96076f-ef17-4054-a346-c348907fe704>
>[2012-04-19 06:53:17.906064] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:17.906429] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:17.962060] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/~WRL0004.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:17.962434] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/HANDOUT.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:17.968091] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/TIPS.DOC on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:18.049581] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>
>[2012-04-19 06:53:18.049609] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:1684b9f9-7891-441a-b0cd-a871152a9439> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:18.049625] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:1684b9f9-7891-441a-b0cd-a871152a9439>
>[2012-04-19 06:53:18.050666] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:59849e5d-6b52-4a30-98ad-45440eef6365>.
>[2012-04-19 06:53:18.050691] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:59849e5d-6b52-4a30-98ad-45440eef6365>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.066473] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/~WRD0000.TMP
>[2012-04-19 06:53:18.175950] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:845c7a84-9eb1-4e44-84e1-27d5f257ad65>
>[2012-04-19 06:53:18.176033] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>
>[2012-04-19 06:53:18.183329] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/PPTC241.TMP 
>[2012-04-19 06:53:18.183354] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/PPTC241.TMP, reason: lookup detected pending operations
>[2012-04-19 06:53:18.183742] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:845c7a84-9eb1-4e44-84e1-27d5f257ad65>
>[2012-04-19 06:53:18.209488] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT
>[2012-04-19 06:53:18.223088] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:d2e52427-a978-47e6-ad62-0d74b0a0ba23>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.223652] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT, reason: lookup detected pending operations
>[2012-04-19 06:53:18.311059] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:8af7c0ac-769f-489d-906b-0ed7157bd77c>/~dmtmp/WORD/~$CHAP10.DOC
>[2012-04-19 06:53:18.328854] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/~WRL0004.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:18.329130] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/~WRL1146.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:18.335577] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:7fdfdb6b-d5ea-4c67-b9cb-64797c7931c7>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:18.353207] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:374b49af-1f0c-4915-bb9b-16fd435beb2c>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.357436] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:a932498a-5c74-4780-8169-88bf3bb17806>
>[2012-04-19 06:53:18.357533] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:4931f1a9-dfcc-43e5-b62f-c7b54f7ff90b>/~WRD0000.TMP
>[2012-04-19 06:53:18.358160] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:18.358722] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:59849e5d-6b52-4a30-98ad-45440eef6365>.
>[2012-04-19 06:53:18.358747] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:59849e5d-6b52-4a30-98ad-45440eef6365>.
>[2012-04-19 06:53:18.361096] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:845c7a84-9eb1-4e44-84e1-27d5f257ad65>
>[2012-04-19 06:53:18.370401] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:371826c7-a2ac-495c-9d85-3e607aec05d3> 
>[2012-04-19 06:53:18.370426] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:371826c7-a2ac-495c-9d85-3e607aec05d3>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.381137] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/TIPS.DOC
>[2012-04-19 06:53:18.381537] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>
>[2012-04-19 06:53:18.390852] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:20b3de70-57df-4fdb-872b-ec0a77b909f5>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.398919] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:ee757995-b923-47b1-9c4f-c84527c56c26>
>[2012-04-19 06:53:18.432755] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:3d243e15-cc83-46c1-bb46-83c5975a04ba> 
>[2012-04-19 06:53:18.432812] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:3d243e15-cc83-46c1-bb46-83c5975a04ba>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.449332] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:3d243e15-cc83-46c1-bb46-83c5975a04ba>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.515614] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:fb73c2aa-af5e-4348-bec4-eb28af0eba75>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:18.530326] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:d8dbc312-f52f-49a0-b85d-7169e4959217>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:18.569991] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:b473fccb-0610-4cef-a1b7-5d2b2d97f2bd>/NEWTIPS.PPT
>[2012-04-19 06:53:18.581374] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:59849e5d-6b52-4a30-98ad-45440eef6365>
>[2012-04-19 06:53:18.581397] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:59849e5d-6b52-4a30-98ad-45440eef6365> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:18.581412] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:59849e5d-6b52-4a30-98ad-45440eef6365>
>[2012-04-19 06:53:18.592872] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-0: entries are missing in lookup of <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>.
>[2012-04-19 06:53:18.592897] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.593190] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>.
>[2012-04-19 06:53:18.593212] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.660320] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:2e8e680e-e409-48c1-88e4-5bac8637cb3c>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:18.666967] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:18.667398] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Non Blocking metadata inodelks failed for <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>.
>[2012-04-19 06:53:18.667419] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-0: Metadata self-heal failed for <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>.
>[2012-04-19 06:53:18.783064] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:ee757995-b923-47b1-9c4f-c84527c56c26>
>[2012-04-19 06:53:18.783278] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory. Path: <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>
>[2012-04-19 06:53:18.783343] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-0: open of <gfid:5633fd1f-949b-453c-a1df-12279204b1e5> failed on child hosdu-client-0 (No such file or directory)
>[2012-04-19 06:53:18.783362] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data entry self-heal failed on <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>
>[2012-04-19 06:53:18.806940] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/NEWPCB.PPT on hosdu-client-1
>[2012-04-19 06:53:18.818833] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:fff94f40-1cd0-4cdb-b280-d0cf467d0e1b>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.833290] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:7fdfdb6b-d5ea-4c67-b9cb-64797c7931c7>
>[2012-04-19 06:53:18.925942] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:4e7d1ad3-0b18-451d-9007-d8eee7168f00>/~dmtmp/WORD
>[2012-04-19 06:53:18.927578] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>
>[2012-04-19 06:53:18.930491] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:12ac305e-7cbd-4df7-804b-679d36645593>
>[2012-04-19 06:53:18.939809] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:5ba808d8-1e6a-4219-9e53-65f18be62e61>, reason: lookup detected pending operations
>[2012-04-19 06:53:18.939952] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:7edff096-5541-43c5-8b08-11a061847d81>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:18.941818] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/TIPS.DOC 
>[2012-04-19 06:53:18.941841] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/TIPS.DOC, reason: lookup detected pending operations
>[2012-04-19 06:53:19.223032] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:b11644e5-3351-4842-a512-7135c71e2894>
>[2012-04-19 06:53:19.224290] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:1401e39f-8e59-43e5-813a-277728bcdefd>
>[2012-04-19 06:53:19.225155] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:352d0e0d-3ebe-49ea-90c1-47bc1ae4a9ae>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:19.225822] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:d58476d6-e07a-479d-b911-0188d317e4ca>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:19.294219] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:19.294652] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>.
>[2012-04-19 06:53:19.294673] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>.
>[2012-04-19 06:53:19.501982] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:511925c2-d287-48b4-9998-3d889326fb92>
>[2012-04-19 06:53:19.629837] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:18fb9c87-df4b-4c06-aa2d-240ad427817b>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.637653] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:18fb9c87-df4b-4c06-aa2d-240ad427817b> 
>[2012-04-19 06:53:19.637677] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:18fb9c87-df4b-4c06-aa2d-240ad427817b>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.686885] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:fb73c2aa-af5e-4348-bec4-eb28af0eba75>
>[2012-04-19 06:53:19.687157] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:d8dbc312-f52f-49a0-b85d-7169e4959217>
>[2012-04-19 06:53:19.691949] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:d2e52427-a978-47e6-ad62-0d74b0a0ba23>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:19.702838] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:41f17448-3355-4e9b-acc2-4591d8c5bf29>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.702914] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:fec37fb6-7fa8-4bdb-b130-ac00d8bb2dcb>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.704713] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:351f0ba7-fc37-4a26-a855-7f3abdd867f1>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.709150] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/PPTC241.TMP 
>[2012-04-19 06:53:19.709649] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/NEWPCB.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:19.761970] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:2e8e680e-e409-48c1-88e4-5bac8637cb3c>
>[2012-04-19 06:53:19.763623] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  entry self-heal triggered. path: <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.764003] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  entry self-heal triggered. path: <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.789815] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:b557406e-553b-4733-99b2-bc40af5ba8ea> 
>[2012-04-19 06:53:19.789844] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:b557406e-553b-4733-99b2-bc40af5ba8ea>, reason: lookup detected pending operations
>[2012-04-19 06:53:19.862819] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>
>[2012-04-19 06:53:19.862847] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:5633fd1f-949b-453c-a1df-12279204b1e5> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:19.862862] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:5633fd1f-949b-453c-a1df-12279204b1e5>
>[2012-04-19 06:53:19.863791] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:9b294302-a673-4a8f-a59a-1bf1108ab6df> 
>[2012-04-19 06:53:19.863823] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:9b294302-a673-4a8f-a59a-1bf1108ab6df>, reason: lookup detected pending operations
>[2012-04-19 06:53:20.047075] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:ee757995-b923-47b1-9c4f-c84527c56c26>
>[2012-04-19 06:53:20.080345] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:0a0c637d-0a87-4825-82bd-9c01c87d993d>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:20.094276] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:f8f6e739-5e04-4b54-9354-ecd3479ab1a8>.
>[2012-04-19 06:53:20.094304] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:f8f6e739-5e04-4b54-9354-ecd3479ab1a8>, reason: lookup detected pending operations
>[2012-04-19 06:53:20.187581] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:1ad6f4b1-7653-4654-acf9-86c3c48c00dc>/~dmtmp/WORD/TIPS.DOC
>[2012-04-19 06:53:20.198064] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:20.198524] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:f8f6e739-5e04-4b54-9354-ecd3479ab1a8>.
>[2012-04-19 06:53:20.198545] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:f8f6e739-5e04-4b54-9354-ecd3479ab1a8>.
>[2012-04-19 06:53:20.199045] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:ee757995-b923-47b1-9c4f-c84527c56c26>
>[2012-04-19 06:53:20.200525] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-1: missing entry <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>/NEWTIPS.PPT on hosdu-client-2
>[2012-04-19 06:53:20.207487] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:f8f6e739-5e04-4b54-9354-ecd3479ab1a8>
>[2012-04-19 06:53:20.207511] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:f8f6e739-5e04-4b54-9354-ecd3479ab1a8> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:20.207525] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:f8f6e739-5e04-4b54-9354-ecd3479ab1a8>
>[2012-04-19 06:53:20.213019] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:511925c2-d287-48b4-9998-3d889326fb92>
>[2012-04-19 06:53:20.213290] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:20.221309] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:200f9cbe-59e0-45ee-9121-54e98efb23c3>, reason: lookup detected pending operations
>[2012-04-19 06:53:20.223945] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:0a0c637d-0a87-4825-82bd-9c01c87d993d>
>[2012-04-19 06:53:20.224044] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:3d243e15-cc83-46c1-bb46-83c5975a04ba>
>[2012-04-19 06:53:20.227905] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:374b49af-1f0c-4915-bb9b-16fd435beb2c>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:20.237968] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:20.238287] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>/PPTC321.TMP on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:20.238374] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-1: path <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>/NEWTIPS.PPT on subvolume hosdu-client-3 => -1 (No such file or directory)
>[2012-04-19 06:53:20.256293] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:63930d14-9e58-4807-b997-42e6fbe547a3> 
>[2012-04-19 06:53:20.256341] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:63930d14-9e58-4807-b997-42e6fbe547a3>, reason: lookup detected pending operations
>[2012-04-19 06:53:20.261891] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:ba53680d-c2f4-43b5-8145-ab7c1b84debc>, reason: lookup detected pending operations
>[2012-04-19 06:53:20.261975] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:20.289375] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:ba53680d-c2f4-43b5-8145-ab7c1b84debc> 
>[2012-04-19 06:53:20.289403] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:ba53680d-c2f4-43b5-8145-ab7c1b84debc>, reason: lookup detected pending operations
>[2012-04-19 06:53:20.304191] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:20.305019] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:20.350399] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:18fb9c87-df4b-4c06-aa2d-240ad427817b>
>[2012-04-19 06:53:20.351866] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:b557406e-553b-4733-99b2-bc40af5ba8ea>
>[2012-04-19 06:53:20.366254] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:20b3de70-57df-4fdb-872b-ec0a77b909f5>: completed. (0 blocks of 3 were different (0.00%))
>[2012-04-19 06:53:20.381516] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:7edff096-5541-43c5-8b08-11a061847d81>
>[2012-04-19 06:53:20.408803] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/PPTC241.TMP
>[2012-04-19 06:53:20.512001] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:497c0414-bddc-482b-a1ba-d579501c7586>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:20.523690] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:352d0e0d-3ebe-49ea-90c1-47bc1ae4a9ae>
>[2012-04-19 06:53:20.523835] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:d58476d6-e07a-479d-b911-0188d317e4ca>
>[2012-04-19 06:53:20.523995] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:371826c7-a2ac-495c-9d85-3e607aec05d3>
>[2012-04-19 06:53:20.524089] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:3d243e15-cc83-46c1-bb46-83c5975a04ba>
>[2012-04-19 06:53:20.527953] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:fff94f40-1cd0-4cdb-b280-d0cf467d0e1b>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:20.547731] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-hosdu-replicate-0: missing entry <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>/NEWPCB.PPT on hosdu-client-1
>[2012-04-19 06:53:20.661746] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT
>[2012-04-19 06:53:20.663708] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/PPTC241.TMP, reason: lookup detected pending operations
>[2012-04-19 06:53:20.880888] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:d2e52427-a978-47e6-ad62-0d74b0a0ba23>
>[2012-04-19 06:53:21.011203] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:9b294302-a673-4a8f-a59a-1bf1108ab6df>
>[2012-04-19 06:53:21.177086] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/TIPS.DOC
>[2012-04-19 06:53:21.178861] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:b421f1f3-d881-4e4d-86c7-60ab93f3e620> 
>[2012-04-19 06:53:21.178887] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:b421f1f3-d881-4e4d-86c7-60ab93f3e620>, reason: lookup detected pending operations
>[2012-04-19 06:53:21.191212] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:b421f1f3-d881-4e4d-86c7-60ab93f3e620>, reason: lookup detected pending operations
>[2012-04-19 06:53:21.227948] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:4c66f717-1fa9-4459-ae6c-175e707fd3ec>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:21.397524] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:41f17448-3355-4e9b-acc2-4591d8c5bf29>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:21.399452] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:fec37fb6-7fa8-4bdb-b130-ac00d8bb2dcb>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:21.415903] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:18fb9c87-df4b-4c06-aa2d-240ad427817b>
>[2012-04-19 06:53:21.474124] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:20b3de70-57df-4fdb-872b-ec0a77b909f5>
>[2012-04-19 06:53:21.486093] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>/NEWPCB.PPT on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:21.488114] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-hosdu-replicate-0: path <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>/PPTC321.TMP on subvolume hosdu-client-0 => -1 (No such file or directory)
>[2012-04-19 06:53:21.498741] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:4c5668ea-2909-49a5-a0c6-ffeef4a90d23>/PPTC241.TMP
>[2012-04-19 06:53:21.505900] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:497c0414-bddc-482b-a1ba-d579501c7586>
>[2012-04-19 06:53:21.507296] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  data self-heal triggered. path: <gfid:48c54738-fc18-4e57-81c4-934a6d303eab>, reason: lookup detected pending operations
>[2012-04-19 06:53:21.508816] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:371826c7-a2ac-495c-9d85-3e607aec05d3>
>[2012-04-19 06:53:21.509049] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:3d243e15-cc83-46c1-bb46-83c5975a04ba>
>[2012-04-19 06:53:21.600672] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:3d243e15-cc83-46c1-bb46-83c5975a04ba>
>[2012-04-19 06:53:21.600822] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:374b49af-1f0c-4915-bb9b-16fd435beb2c>
>[2012-04-19 06:53:21.617818] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:fff94f40-1cd0-4cdb-b280-d0cf467d0e1b>
>[2012-04-19 06:53:21.629961] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:200f9cbe-59e0-45ee-9121-54e98efb23c3>: completed. (0 blocks of 2 were different (0.00%))
>[2012-04-19 06:53:21.634762] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  entry self-heal completed on <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>
>[2012-04-19 06:53:21.984967] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:18fb9c87-df4b-4c06-aa2d-240ad427817b>
>[2012-04-19 06:53:21.985201] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:b557406e-553b-4733-99b2-bc40af5ba8ea>
>[2012-04-19 06:53:21.985851] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:9b294302-a673-4a8f-a59a-1bf1108ab6df>
>[2012-04-19 06:53:21.986846] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-1: diff self-heal on <gfid:5c02d1bd-5fad-4388-ab67-9e56de3a324a>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:21.987300] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:ba53680d-c2f4-43b5-8145-ab7c1b84debc>
>[2012-04-19 06:53:21.990943] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/PPTC241.TMP
>[2012-04-19 06:53:21.999358] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:b421f1f3-d881-4e4d-86c7-60ab93f3e620>
>[2012-04-19 06:53:22.003149] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  data self-heal completed on <gfid:5c02d1bd-5fad-4388-ab67-9e56de3a324a>
>[2012-04-19 06:53:22.005122] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:ba53680d-c2f4-43b5-8145-ab7c1b84debc>
>[2012-04-19 06:53:22.010813] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:907de290-4315-4858-8dda-22982a0b3122>/~dmtmp/PWRPNT/PPTC241.TMP
>[2012-04-19 06:53:22.014869] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:db638c84-1efb-4ef0-bd3b-f847785686a5> 
>[2012-04-19 06:53:22.014894] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:db638c84-1efb-4ef0-bd3b-f847785686a5>, reason: lookup detected pending operations
>[2012-04-19 06:53:22.018853] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:b421f1f3-d881-4e4d-86c7-60ab93f3e620>
>[2012-04-19 06:53:22.184864] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:63930d14-9e58-4807-b997-42e6fbe547a3>
>[2012-04-19 06:53:22.185124] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:ba53680d-c2f4-43b5-8145-ab7c1b84debc>
>[2012-04-19 06:53:22.201059] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:f36dfc41-8fa4-4d69-995e-fd68e3c2e17f>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:22.239742] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:99e7dbd7-aea9-44cb-a98e-19d16db716f8>/TIPS.DOC
>[2012-04-19 06:53:22.296469] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  entry self-heal completed on <gfid:baac4621-57b6-420c-b6d1-98e9f3322b0f>
>[2012-04-19 06:53:22.351162] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:8c680689-8eac-40da-9724-ed739bbee102>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:22.365337] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:4c66f717-1fa9-4459-ae6c-175e707fd3ec>
>[2012-04-19 06:53:22.392740] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:41f17448-3355-4e9b-acc2-4591d8c5bf29>
>[2012-04-19 06:53:22.415837] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:fec37fb6-7fa8-4bdb-b130-ac00d8bb2dcb>
>[2012-04-19 06:53:22.417495] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:18fb9c87-df4b-4c06-aa2d-240ad427817b>
>[2012-04-19 06:53:22.417767] I [afr-common.c:1209:afr_detect_self_heal_by_lookup_status] 0-hosdu-replicate-1: entries are missing in lookup of <gfid:8aa37628-0a9a-4dd5-bb3a-adc8f6413000>.
>[2012-04-19 06:53:22.417816] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data entry missing-entry gfid self-heal triggered. path: <gfid:8aa37628-0a9a-4dd5-bb3a-adc8f6413000>, reason: lookup detected pending operations
>[2012-04-19 06:53:22.428988] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:5ba808d8-1e6a-4219-9e53-65f18be62e61>: completed. (0 blocks of 8 were different (0.00%))
>[2012-04-19 06:53:22.516220] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:b421f1f3-d881-4e4d-86c7-60ab93f3e620>
>[2012-04-19 06:53:22.519881] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:200f9cbe-59e0-45ee-9121-54e98efb23c3>
>[2012-04-19 06:53:22.663034] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:63930d14-9e58-4807-b997-42e6fbe547a3>
>[2012-04-19 06:53:22.668378] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:2fb593ba-abe6-482f-bebc-dc014eeb4321> 
>[2012-04-19 06:53:22.668405] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:2fb593ba-abe6-482f-bebc-dc014eeb4321>, reason: lookup detected pending operations
>[2012-04-19 06:53:22.674958] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:ba53680d-c2f4-43b5-8145-ab7c1b84debc>
>[2012-04-19 06:53:22.675671] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:f36dfc41-8fa4-4d69-995e-fd68e3c2e17f>
>[2012-04-19 06:53:22.676643] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:8c680689-8eac-40da-9724-ed739bbee102>
>[2012-04-19 06:53:22.708208] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:749b12db-cc63-4965-a835-03f81aec48d4> 
>[2012-04-19 06:53:22.708240] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:749b12db-cc63-4965-a835-03f81aec48d4>, reason: lookup detected pending operations
>[2012-04-19 06:53:23.024040] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-1: size differs for <gfid:ccc03524-5761-48ab-94a8-85198fa425c1> 
>[2012-04-19 06:53:23.024094] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-1: background  meta-data data self-heal triggered. path: <gfid:ccc03524-5761-48ab-94a8-85198fa425c1>, reason: lookup detected pending operations
>[2012-04-19 06:53:23.034740] W [client3_1-fops.c:1489:client3_1_inodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:23.035203] E [afr-self-heal-metadata.c:548:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Non Blocking metadata inodelks failed for <gfid:8aa37628-0a9a-4dd5-bb3a-adc8f6413000>.
>[2012-04-19 06:53:23.035232] E [afr-self-heal-metadata.c:550:afr_sh_metadata_post_nonblocking_inodelk_cbk] 0-hosdu-replicate-1: Metadata self-heal failed for <gfid:8aa37628-0a9a-4dd5-bb3a-adc8f6413000>.
>[2012-04-19 06:53:23.047028] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:23.048183] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:23.048638] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-0: remote operation failed: No such file or directory
>[2012-04-19 06:53:23.075433] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:23.077543] W [client3_1-fops.c:1539:client3_1_finodelk_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:23.077938] W [client3_1-fops.c:882:client3_1_flush_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory
>[2012-04-19 06:53:23.142891] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:5ba808d8-1e6a-4219-9e53-65f18be62e61>
>[2012-04-19 06:53:23.201887] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:b421f1f3-d881-4e4d-86c7-60ab93f3e620>
>[2012-04-19 06:53:23.203527] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:db638c84-1efb-4ef0-bd3b-f847785686a5>
>[2012-04-19 06:53:23.480316] W [client3_1-fops.c:419:client3_1_open_cbk] 0-hosdu-client-3: remote operation failed: No such file or directory. Path: <gfid:8aa37628-0a9a-4dd5-bb3a-adc8f6413000>
>[2012-04-19 06:53:23.480357] E [afr-self-heal-data.c:1330:afr_sh_data_open_cbk] 0-hosdu-replicate-1: open of <gfid:8aa37628-0a9a-4dd5-bb3a-adc8f6413000> failed on child hosdu-client-3 (No such file or directory)
>[2012-04-19 06:53:23.480375] E [afr-self-heal-common.c:2042:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data entry self-heal failed on <gfid:8aa37628-0a9a-4dd5-bb3a-adc8f6413000>
>[2012-04-19 06:53:23.729868] I [afr-common.c:1183:afr_detect_self_heal_by_iatt] 0-hosdu-replicate-0: size differs for <gfid:925d8874-d058-457f-a326-62587ed8490d> 
>[2012-04-19 06:53:23.729918] I [afr-common.c:1335:afr_launch_self_heal] 0-hosdu-replicate-0: background  meta-data data self-heal triggered. path: <gfid:925d8874-d058-457f-a326-62587ed8490d>, reason: lookup detected pending operations
>[2012-04-19 06:53:23.732089] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:2fb593ba-abe6-482f-bebc-dc014eeb4321>
>[2012-04-19 06:53:23.739836] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:749b12db-cc63-4965-a835-03f81aec48d4>
>[2012-04-19 06:53:23.741945] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-1: full self-heal completed on <gfid:ccc03524-5761-48ab-94a8-85198fa425c1>
>[2012-04-19 06:53:23.755873] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:2fb593ba-abe6-482f-bebc-dc014eeb4321>
>[2012-04-19 06:53:23.755951] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:749b12db-cc63-4965-a835-03f81aec48d4>
>[2012-04-19 06:53:23.759829] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-1: background  meta-data data self-heal completed on <gfid:ccc03524-5761-48ab-94a8-85198fa425c1>
>[2012-04-19 06:53:24.860036] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:db638c84-1efb-4ef0-bd3b-f847785686a5>
>[2012-04-19 06:53:25.685810] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:351f0ba7-fc37-4a26-a855-7f3abdd867f1>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:27.241832] I [afr-self-heal-algorithm.c:125:sh_loop_driver_done] 0-hosdu-replicate-0: full self-heal completed on <gfid:925d8874-d058-457f-a326-62587ed8490d>
>[2012-04-19 06:53:27.241972] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:351f0ba7-fc37-4a26-a855-7f3abdd867f1>
>[2012-04-19 06:53:27.997888] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-hosdu-replicate-0: diff self-heal on <gfid:48c54738-fc18-4e57-81c4-934a6d303eab>: completed. (0 blocks of 15 were different (0.00%))
>[2012-04-19 06:53:28.265706] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  meta-data data self-heal completed on <gfid:925d8874-d058-457f-a326-62587ed8490d>
>[2012-04-19 06:53:28.268108] I [afr-self-heal-common.c:2045:afr_self_heal_completion_cbk] 0-hosdu-replicate-0: background  data self-heal completed on <gfid:48c54738-fc18-4e57-81c4-934a6d303eab>

Actions: View

Attachments on bug 802779: 569688 | 578632