Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 621592 Details for
Bug 863084
"gluster volume heal <vol_name> info healed" command execution unsuccessful
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
glustershd log file.
glustershd.log (text/x-log), 2.32 MB, created by
spandura
on 2012-10-04 12:06:27 UTC
(
hide
)
Description:
glustershd log file.
Filename:
MIME Type:
Creator:
spandura
Created:
2012-10-04 12:06:27 UTC
Size:
2.32 MB
patch
obsolete
>[2012-09-30 03:38:01.395085] I [glusterfsd-mgmt.c:1568:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2012-09-30 03:38:18.637129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 03:38:18.637182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 03:38:18.637249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:34:53.014729 (xid=0x689x) >[2012-09-30 03:38:18.637270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 03:38:18.637286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 03:41:04.069796] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 03:41:04.072959] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 03:41:04.075459] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 03:41:45.667119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 03:41:45.667167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 03:41:45.667233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:38:20.044282 (xid=0x690x) >[2012-09-30 03:41:45.667253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 03:41:45.667269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 03:45:12.700134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 03:45:12.700183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 03:45:12.700249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:41:47.077314 (xid=0x691x) >[2012-09-30 03:45:12.700271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 03:45:12.700289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 03:48:39.729138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 03:48:39.729185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 03:48:39.729264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:45:14.106801 (xid=0x692x) >[2012-09-30 03:48:39.729285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 03:48:39.729302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 03:51:04.158948] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 03:51:04.161963] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 03:51:04.164218] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 03:52:06.759119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 03:52:06.759168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 03:52:06.759233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:48:41.136745 (xid=0x693x) >[2012-09-30 03:52:06.759254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 03:52:06.759269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 03:55:33.792107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 03:55:33.792156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 03:55:33.792220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:52:08.169180 (xid=0x694x) >[2012-09-30 03:55:33.792242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 03:55:33.792258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 03:59:00.822131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 03:59:00.822180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 03:59:00.822246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:55:35.199057 (xid=0x695x) >[2012-09-30 03:59:00.822277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 03:59:00.822295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:01:04.246939] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:01:04.249493] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:01:04.251757] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:02:27.851147] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:02:27.851199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:02:27.851265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 03:59:02.228218 (xid=0x696x) >[2012-09-30 04:02:27.851286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:02:27.851302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:05:54.883109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:05:54.883160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:05:54.883226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:02:29.260073 (xid=0x697x) >[2012-09-30 04:05:54.883247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:05:54.883264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:09:21.912114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:09:21.912163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:09:21.912229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:05:56.289299 (xid=0x698x) >[2012-09-30 04:09:21.912250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:09:21.912267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:11:04.334894] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:11:04.337722] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:11:04.340073] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:12:48.941143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:12:48.941192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:12:48.941257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:09:23.318748 (xid=0x699x) >[2012-09-30 04:12:48.941278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:12:48.941295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:16:15.971139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:16:15.971188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:16:15.971254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:12:50.348386 (xid=0x700x) >[2012-09-30 04:16:15.971275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:16:15.971291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:19:43.000156] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:19:43.000208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:19:43.000275] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:16:17.377402 (xid=0x701x) >[2012-09-30 04:19:43.000297] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:19:43.000314] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:21:04.418669] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:21:04.421792] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:21:04.424242] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:23:10.028130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:23:10.028180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:23:10.028256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:19:44.405394 (xid=0x702x) >[2012-09-30 04:23:10.029005] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:23:10.029040] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:26:37.060146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:26:37.060196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:26:37.060262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:23:11.436846 (xid=0x703x) >[2012-09-30 04:26:37.060283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:26:37.060300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:30:04.088145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:30:04.088193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:30:04.088261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:26:38.465360 (xid=0x704x) >[2012-09-30 04:30:04.088282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:30:04.088298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:31:04.505294] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:31:04.508238] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:31:04.510535] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:33:31.117117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:33:31.117165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:33:31.117231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:30:05.493911 (xid=0x705x) >[2012-09-30 04:33:31.117252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:33:31.117268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:36:58.148108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:36:58.148159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:36:58.148225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:33:32.525310 (xid=0x706x) >[2012-09-30 04:36:58.148247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:36:58.148264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:40:25.178120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:40:25.178175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:40:25.178242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:36:59.554860 (xid=0x707x) >[2012-09-30 04:40:25.178264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:40:25.178280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:41:04.593730] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:41:04.596554] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:41:04.598762] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:43:52.207126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:43:52.207185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:43:52.207259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:40:26.584332 (xid=0x708x) >[2012-09-30 04:43:52.208314] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:43:52.208350] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:47:19.241170] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:47:19.241222] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:47:19.241293] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:43:53.618002 (xid=0x709x) >[2012-09-30 04:47:19.241321] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:47:19.241339] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:50:46.271169] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:50:46.271220] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:50:46.271288] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:47:20.647818 (xid=0x710x) >[2012-09-30 04:50:46.271309] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:50:46.271325] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:51:04.682298] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:51:04.685145] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:51:04.687338] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 04:54:13.299112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:54:13.299161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:54:13.299228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:50:47.676234 (xid=0x711x) >[2012-09-30 04:54:13.299263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:54:13.299280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 04:57:40.331170] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 04:57:40.331220] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 04:57:40.331286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:54:14.708651 (xid=0x712x) >[2012-09-30 04:57:40.331307] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 04:57:40.331325] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:01:04.765139] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:01:04.767789] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:01:04.770061] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:01:07.360100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:01:07.360141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:01:07.360209] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 04:57:41.737070 (xid=0x713x) >[2012-09-30 05:01:07.360230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:01:07.360245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:04:34.389115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:04:34.389169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:04:34.389236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:01:08.766557 (xid=0x714x) >[2012-09-30 05:04:34.389257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:04:34.389284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:08:01.420120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:08:01.420175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:08:01.420246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:04:35.797311 (xid=0x715x) >[2012-09-30 05:08:01.420268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:08:01.420284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:11:04.851785] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:11:04.854853] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:11:04.857057] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:11:28.449134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:11:28.449179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:11:28.449245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:08:02.826049 (xid=0x716x) >[2012-09-30 05:11:28.449265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:11:28.449280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:14:55.479089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:14:55.479136] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:14:55.479203] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:11:29.856020 (xid=0x717x) >[2012-09-30 05:14:55.479224] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:14:55.479240] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:18:22.816114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:18:22.816167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:18:22.816244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:14:57.193303 (xid=0x718x) >[2012-09-30 05:18:22.816265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:18:22.816281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:21:05.251383] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:21:05.254140] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:21:05.256420] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:21:49.848088] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:21:49.848130] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:21:49.848194] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:18:24.225485 (xid=0x719x) >[2012-09-30 05:21:49.848215] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:21:49.848231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:25:16.881092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:25:16.881145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:25:16.881212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:21:51.258770 (xid=0x720x) >[2012-09-30 05:25:16.881234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:25:16.881251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:28:43.915092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:28:43.915141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:28:43.915208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:25:18.292262 (xid=0x721x) >[2012-09-30 05:28:43.915240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:28:43.915257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:31:05.346393] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:31:05.349252] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:31:05.351428] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:32:10.947095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:32:10.947140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:32:10.947203] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:28:45.324133 (xid=0x722x) >[2012-09-30 05:32:10.947224] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:32:10.947240] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:35:37.980116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:35:37.980165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:35:37.980233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:32:12.357345 (xid=0x723x) >[2012-09-30 05:35:37.980255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:35:37.980272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:39:05.010135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:39:05.010186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:39:05.010262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:35:39.387415 (xid=0x724x) >[2012-09-30 05:39:05.010292] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:39:05.010311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:41:05.434313] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:41:05.437367] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:41:05.439694] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:42:32.039117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:42:32.039168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:42:32.039236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:39:06.415848 (xid=0x725x) >[2012-09-30 05:42:32.039257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:42:32.039275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:45:59.069128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:45:59.069176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:45:59.069241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:42:33.446561 (xid=0x726x) >[2012-09-30 05:45:59.069262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:45:59.069278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:49:26.097077] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:49:26.097129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:49:26.097194] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:46:00.473828 (xid=0x727x) >[2012-09-30 05:49:26.097215] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:49:26.097232] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:51:05.517390] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:51:05.519779] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:51:05.522291] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 05:52:53.125118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:52:53.125163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:52:53.125228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:49:27.502012 (xid=0x728x) >[2012-09-30 05:52:53.125249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:52:53.125265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:56:20.156115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:56:20.156164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:56:20.156232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:52:54.533042 (xid=0x729x) >[2012-09-30 05:56:20.156254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:56:20.156271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 05:59:47.188111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 05:59:47.188162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 05:59:47.188227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:56:21.565535 (xid=0x730x) >[2012-09-30 05:59:47.188249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 05:59:47.188266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:01:05.610864] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:01:05.613738] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:01:05.615956] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:03:14.219106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:03:14.219159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:03:14.219226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 05:59:48.596659 (xid=0x731x) >[2012-09-30 06:03:14.219248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:03:14.219265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:06:41.253120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:06:41.253169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:06:41.253235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:03:15.630653 (xid=0x732x) >[2012-09-30 06:06:41.253256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:06:41.253273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:10:08.287093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:10:08.287142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:10:08.287211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:06:42.664367 (xid=0x733x) >[2012-09-30 06:10:08.287233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:10:08.287250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:11:05.704816] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:11:05.707846] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:11:05.710388] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:13:35.317120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:13:35.317182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:13:35.317259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:10:09.693828 (xid=0x734x) >[2012-09-30 06:13:35.318248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:13:35.318291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:17:02.348125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:17:02.348185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:17:02.348269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:13:36.725798 (xid=0x735x) >[2012-09-30 06:17:02.349063] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:17:02.349106] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:20:29.381106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:20:29.381157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:20:29.381225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:17:03.758764 (xid=0x736x) >[2012-09-30 06:20:29.381247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:20:29.381264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:21:05.797386] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:21:05.800270] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:21:05.802491] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:23:56.412133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:23:56.412187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:23:56.412253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:20:30.789024 (xid=0x737x) >[2012-09-30 06:23:56.412284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:23:56.412302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:27:23.444102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:27:23.444151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:27:23.444215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:23:57.821401 (xid=0x738x) >[2012-09-30 06:27:23.444237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:27:23.444253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:30:50.477143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:30:50.477194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:30:50.477261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:27:24.854291 (xid=0x739x) >[2012-09-30 06:30:50.477282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:30:50.477299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:31:05.892275] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:31:05.895171] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:31:05.897559] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:34:17.508124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:34:17.508173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:34:17.508240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:30:51.885245 (xid=0x740x) >[2012-09-30 06:34:17.508261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:34:17.508289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:37:44.542123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:37:44.542178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:37:44.542243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:34:18.919445 (xid=0x741x) >[2012-09-30 06:37:44.542265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:37:44.542282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:41:05.983832] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:41:05.987061] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:41:05.989483] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:41:11.576125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:41:11.576169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:41:11.576237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:37:45.952883 (xid=0x742x) >[2012-09-30 06:41:11.576258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:41:11.576274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:44:38.610135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:44:38.610187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:44:38.610253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:41:12.987247 (xid=0x743x) >[2012-09-30 06:44:38.610275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:44:38.610292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:48:05.642134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:48:05.642190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:48:05.642277] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:44:40.019275 (xid=0x744x) >[2012-09-30 06:48:05.642299] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:48:05.642315] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:51:06.080820] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:51:06.084156] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:51:06.086562] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 06:51:32.674118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:51:32.674162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:51:32.674227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:48:07.051686 (xid=0x745x) >[2012-09-30 06:51:32.674248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:51:32.674265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:54:59.710132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:54:59.710181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:54:59.710247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:51:34.087311 (xid=0x746x) >[2012-09-30 06:54:59.710268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:54:59.710285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 06:58:26.744111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 06:58:26.744162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 06:58:26.744227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:55:01.120837 (xid=0x747x) >[2012-09-30 06:58:26.744259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 06:58:26.744277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:01:06.183312] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:01:06.185808] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:01:06.188030] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:01:53.779113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:01:53.779160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:01:53.779226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 06:58:28.156029 (xid=0x748x) >[2012-09-30 07:01:53.779247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:01:53.779262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:05:20.814124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:05:20.814176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:05:20.814251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:01:55.191384 (xid=0x749x) >[2012-09-30 07:05:20.815218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:05:20.815254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:08:47.848147] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:08:47.848199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:08:47.848266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:05:22.225766 (xid=0x750x) >[2012-09-30 07:08:47.848288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:08:47.848314] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:11:06.286398] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:11:06.288963] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:11:06.291208] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:12:14.885125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:12:14.885177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:12:14.885243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:08:49.262133 (xid=0x751x) >[2012-09-30 07:12:14.885264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:12:14.885280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:15:41.921130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:15:41.921178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:15:41.921244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:12:16.298069 (xid=0x752x) >[2012-09-30 07:15:41.921265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:15:41.921282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:19:08.956134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:19:08.956186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:19:08.956254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:15:43.333361 (xid=0x753x) >[2012-09-30 07:19:08.956275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:19:08.956291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:21:06.692276] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:21:06.695373] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:21:06.697856] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:22:36.298111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:22:36.298158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:22:36.298222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:19:10.675262 (xid=0x754x) >[2012-09-30 07:22:36.298244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:22:36.298260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:26:03.327154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:26:03.327204] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:26:03.327278] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:22:37.704201 (xid=0x755x) >[2012-09-30 07:26:03.328227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:26:03.328269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:29:30.353177] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:29:30.353226] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:29:30.353293] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:26:04.730660 (xid=0x756x) >[2012-09-30 07:29:30.353316] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:29:30.353333] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:31:06.773979] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:31:06.776783] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:31:06.778958] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:32:57.382165] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:32:57.382215] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:32:57.382281] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:29:31.759307 (xid=0x757x) >[2012-09-30 07:32:57.382302] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:32:57.382318] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:36:24.410166] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:36:24.410220] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:36:24.410287] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:32:58.787452 (xid=0x758x) >[2012-09-30 07:36:24.410308] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:36:24.410325] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:39:51.437109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:39:51.437157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:39:51.437222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:36:25.814175 (xid=0x759x) >[2012-09-30 07:39:51.437244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:39:51.437260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:41:06.855314] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:41:06.858579] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:41:06.861283] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:43:18.464177] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:43:18.464236] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:43:18.464304] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:39:52.842046 (xid=0x760x) >[2012-09-30 07:43:18.464325] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:43:18.464343] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:46:45.495092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:46:45.495144] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:46:45.495209] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:43:19.872109 (xid=0x761x) >[2012-09-30 07:46:45.495230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:46:45.495248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:50:12.522163] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:50:12.522212] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:50:12.522280] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:46:46.899155 (xid=0x762x) >[2012-09-30 07:50:12.522303] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:50:12.522319] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:51:06.938538] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:51:06.942011] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:51:06.944849] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 07:53:39.551139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:53:39.551189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:53:39.551255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:50:13.928231 (xid=0x763x) >[2012-09-30 07:53:39.551287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:53:39.551304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 07:57:06.579115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 07:57:06.579169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 07:57:06.579236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:53:40.956101 (xid=0x764x) >[2012-09-30 07:57:06.579258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 07:57:06.579275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:00:33.608135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:00:33.608185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:00:33.608252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 07:57:07.985433 (xid=0x765x) >[2012-09-30 08:00:33.608273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:00:33.608290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:01:07.022962] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:01:07.026195] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:01:07.028539] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:04:00.636174] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:04:00.636221] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:04:00.636288] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:00:35.014024 (xid=0x766x) >[2012-09-30 08:04:00.636320] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:04:00.636338] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:07:27.668106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:07:27.668159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:07:27.668226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:04:02.045010 (xid=0x767x) >[2012-09-30 08:07:27.668248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:07:27.668264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:10:54.694078] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:10:54.694131] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:10:54.694198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:07:29.071966 (xid=0x768x) >[2012-09-30 08:10:54.694219] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:10:54.694236] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:11:07.111584] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:11:07.114520] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:11:07.116844] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:14:21.728109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:14:21.728158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:14:21.728225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:10:56.105175 (xid=0x769x) >[2012-09-30 08:14:21.728246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:14:21.728262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:17:48.762118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:17:48.762178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:17:48.762245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:14:23.139101 (xid=0x770x) >[2012-09-30 08:17:48.762266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:17:48.762282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:21:07.195967] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:21:07.199243] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:21:07.201476] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:21:15.789131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:21:15.789174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:21:15.789243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:17:50.166486 (xid=0x771x) >[2012-09-30 08:21:15.789264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:21:15.789281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:24:42.824138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:24:42.824190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:24:42.824254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:21:17.201564 (xid=0x772x) >[2012-09-30 08:24:42.824275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:24:42.824292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:28:09.851081] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:28:09.851137] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:28:09.851212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:24:44.228584 (xid=0x773x) >[2012-09-30 08:28:09.851246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:28:09.851265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:31:07.286503] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:31:07.289592] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:31:07.291899] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:31:36.882096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:31:36.882141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:31:36.882205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:28:11.259129 (xid=0x774x) >[2012-09-30 08:31:36.882225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:31:36.882241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:35:03.917142] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:35:03.917193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:35:03.917259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:31:38.294688 (xid=0x775x) >[2012-09-30 08:35:03.917280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:35:03.917296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:38:30.944109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:38:30.944158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:38:30.944224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:35:05.321997 (xid=0x776x) >[2012-09-30 08:38:30.944245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:38:30.944273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:41:07.373392] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:41:07.376394] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:41:07.378552] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:41:57.975098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:41:57.975145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:41:57.975208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:38:32.352420 (xid=0x777x) >[2012-09-30 08:41:57.975229] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:41:57.975245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:45:25.006145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:45:25.006196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:45:25.006263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:41:59.383749 (xid=0x778x) >[2012-09-30 08:45:25.006284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:45:25.006300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:48:52.035103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:48:52.035150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:48:52.035215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:45:26.412520 (xid=0x779x) >[2012-09-30 08:48:52.035236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:48:52.035253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:51:07.462350] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:51:07.465297] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:51:07.467491] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 08:52:19.063151] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:52:19.063206] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:52:19.063276] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:48:53.440023 (xid=0x780x) >[2012-09-30 08:52:19.063298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:52:19.063315] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:55:46.097145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:55:46.097199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:55:46.097268] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:52:20.474434 (xid=0x781x) >[2012-09-30 08:55:46.097289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:55:46.097305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 08:59:13.124124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 08:59:13.124173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 08:59:13.124241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:55:47.501599 (xid=0x782x) >[2012-09-30 08:59:13.124262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 08:59:13.124279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:01:07.546969] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:01:07.549701] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:01:07.551893] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:02:40.153159] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:02:40.153205] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:02:40.153270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 08:59:14.530824 (xid=0x783x) >[2012-09-30 09:02:40.153291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:02:40.153307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:06:07.184157] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:06:07.184207] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:06:07.184273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:02:41.561972 (xid=0x784x) >[2012-09-30 09:06:07.184294] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:06:07.184310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:09:34.212115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:09:34.212168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:09:34.212238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:06:08.589708 (xid=0x785x) >[2012-09-30 09:09:34.212260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:09:34.212276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:11:07.632629] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:11:07.635389] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:11:07.637781] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:13:01.239167] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:13:01.239219] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:13:01.239286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:09:35.616964 (xid=0x786x) >[2012-09-30 09:13:01.239307] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:13:01.239324] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:16:28.279129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:16:28.279179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:16:28.279246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:13:02.656040 (xid=0x787x) >[2012-09-30 09:16:28.279267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:16:28.279284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:19:55.305150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:19:55.305199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:19:55.305266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:16:29.682485 (xid=0x788x) >[2012-09-30 09:19:55.305287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:19:55.305305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:21:07.721651] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:21:07.725121] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:21:07.727886] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:23:22.332155] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:23:22.332203] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:23:22.332270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:19:56.709508 (xid=0x789x) >[2012-09-30 09:23:22.332303] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:23:22.332320] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:26:49.363110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:26:49.363162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:26:49.363229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:23:23.740661 (xid=0x790x) >[2012-09-30 09:26:49.363250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:26:49.363267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:30:16.390092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:30:16.390151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:30:16.390224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:26:50.767756 (xid=0x791x) >[2012-09-30 09:30:16.390246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:30:16.390262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:31:07.804161] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:31:07.807500] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:31:07.809949] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:33:43.417102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:33:43.417151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:33:43.417219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:30:17.794781 (xid=0x792x) >[2012-09-30 09:33:43.417250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:33:43.417268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:37:10.448088] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:37:10.448141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:37:10.448219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:33:44.825176 (xid=0x793x) >[2012-09-30 09:37:10.448882] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:37:10.448922] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:40:37.475109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:40:37.475159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:40:37.475226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:37:11.852569 (xid=0x794x) >[2012-09-30 09:40:37.475247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:40:37.475264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:41:07.887188] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:41:07.890335] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:41:07.892383] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:44:04.502104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:44:04.502157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:44:04.502225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:40:38.879814 (xid=0x795x) >[2012-09-30 09:44:04.502245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:44:04.502262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:47:31.843106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:47:31.843165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:47:31.843232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:44:06.220593 (xid=0x796x) >[2012-09-30 09:47:31.843252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:47:31.843268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:50:58.874136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:50:58.874185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:50:58.874252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:47:33.251237 (xid=0x797x) >[2012-09-30 09:50:58.874273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:50:58.874290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:51:08.288860] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:51:08.292277] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:51:08.295021] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 09:54:25.905110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:54:25.905165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:54:25.905229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:51:00.282310 (xid=0x798x) >[2012-09-30 09:54:25.905250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:54:25.905267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 09:57:52.940106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 09:57:52.940154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 09:57:52.940221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:54:27.317548 (xid=0x799x) >[2012-09-30 09:57:52.940253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 09:57:52.940270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:01:08.375171] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:01:08.378503] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:01:08.381032] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:01:19.970161] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:01:19.970208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:01:19.970284] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 09:57:54.346923 (xid=0x800x) >[2012-09-30 10:01:19.970306] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:01:19.970323] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:04:47.002131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:04:47.002184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:04:47.002251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:01:21.379024 (xid=0x801x) >[2012-09-30 10:04:47.002272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:04:47.002288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:08:14.340098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:08:14.340154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:08:14.340229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:04:48.717157 (xid=0x802x) >[2012-09-30 10:08:14.340251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:08:14.340283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:11:08.770710] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:11:08.773258] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:11:08.775404] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:11:41.369102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:11:41.369146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:11:41.369210] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:08:15.746048 (xid=0x803x) >[2012-09-30 10:11:41.369231] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:11:41.369247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:15:08.399113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:15:08.399163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:15:08.399230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:11:42.775982 (xid=0x804x) >[2012-09-30 10:15:08.399251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:15:08.399268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:18:35.428097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:18:35.428150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:18:35.428217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:15:09.804896 (xid=0x805x) >[2012-09-30 10:18:35.428239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:18:35.428255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:21:08.857369] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:21:08.860083] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:21:08.862337] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:22:02.457124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:22:02.457173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:22:02.457238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:18:36.834567 (xid=0x806x) >[2012-09-30 10:22:02.457260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:22:02.457275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:25:29.493101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:25:29.493149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:25:29.493216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:22:03.870416 (xid=0x807x) >[2012-09-30 10:25:29.493238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:25:29.493255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:28:56.523117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:28:56.523168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:28:56.523235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:25:30.900562 (xid=0x808x) >[2012-09-30 10:28:56.523256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:28:56.523273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:31:08.951951] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:31:08.954797] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:31:08.957033] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:32:23.553987] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:32:23.554034] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:32:23.554101] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:28:57.931022 (xid=0x809x) >[2012-09-30 10:32:23.554122] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:32:23.554138] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:35:50.585093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:35:50.585140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:35:50.585204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:32:24.962772 (xid=0x810x) >[2012-09-30 10:35:50.585226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:35:50.585243] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:39:17.616112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:39:17.616172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:39:17.616247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:35:51.993631 (xid=0x811x) >[2012-09-30 10:39:17.616269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:39:17.616286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:41:09.042927] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:41:09.045719] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:41:09.048001] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:42:44.647105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:42:44.647153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:42:44.647220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:39:19.024647 (xid=0x812x) >[2012-09-30 10:42:44.647241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:42:44.647258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:46:11.679120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:46:11.679170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:46:11.679235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:42:46.056398 (xid=0x813x) >[2012-09-30 10:46:11.679256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:46:11.679273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:49:38.709100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:49:38.709154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:49:38.709224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:46:13.086654 (xid=0x814x) >[2012-09-30 10:49:38.709246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:49:38.709262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:51:09.136526] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:51:09.139494] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:51:09.141830] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 10:53:05.742118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:53:05.742166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:53:05.742243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:49:40.119389 (xid=0x815x) >[2012-09-30 10:53:05.742264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:53:05.742280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:56:32.776123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:56:32.776171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:56:32.776239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:53:07.153563 (xid=0x816x) >[2012-09-30 10:56:32.776260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:56:32.776277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 10:59:59.811109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 10:59:59.811159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 10:59:59.811225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 10:56:34.188145 (xid=0x817x) >[2012-09-30 10:59:59.811247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 10:59:59.811263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:01:09.235252] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:01:09.238101] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:01:09.240275] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:03:26.844066] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:03:26.844118] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:03:26.844185] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:00:01.221590 (xid=0x818x) >[2012-09-30 11:03:26.844218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:03:26.844235] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:06:53.878070] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:06:53.878120] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:06:53.878187] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:03:28.255714 (xid=0x819x) >[2012-09-30 11:06:53.878208] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:06:53.878225] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:10:20.914114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:10:20.914167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:10:20.914240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:06:55.291681 (xid=0x820x) >[2012-09-30 11:10:20.914262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:10:20.914280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:11:09.336032] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:11:09.338921] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:11:09.341196] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:13:47.948113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:13:47.948164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:13:47.948230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:10:22.325264 (xid=0x821x) >[2012-09-30 11:13:47.948252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:13:47.948268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:17:14.982109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:17:14.982180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:17:14.982261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:13:49.359243 (xid=0x822x) >[2012-09-30 11:17:14.983093] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:17:14.983144] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:20:42.016120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:20:42.016169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:20:42.016235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:17:16.393038 (xid=0x823x) >[2012-09-30 11:20:42.016257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:20:42.016274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:21:09.431255] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:21:09.433859] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:21:09.436168] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:24:09.047119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:24:09.047167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:24:09.047234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:20:43.424078 (xid=0x824x) >[2012-09-30 11:24:09.047255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:24:09.047272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:27:36.079108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:27:36.079159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:27:36.079224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:24:10.456374 (xid=0x825x) >[2012-09-30 11:27:36.079256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:27:36.079273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:31:03.111104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:31:03.111155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:31:03.111219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:27:37.488073 (xid=0x826x) >[2012-09-30 11:31:03.111240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:31:03.111257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:31:09.522783] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:31:09.525534] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:31:09.527882] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:34:30.141109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:34:30.141160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:34:30.141236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:31:04.518370 (xid=0x827x) >[2012-09-30 11:34:30.142645] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:34:30.142681] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:37:57.173106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:37:57.173154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:37:57.173220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:34:31.550303 (xid=0x828x) >[2012-09-30 11:37:57.173252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:37:57.173269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:41:09.612918] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:41:09.616237] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:41:09.618621] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:41:24.206107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:41:24.206151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:41:24.206215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:37:58.583624 (xid=0x829x) >[2012-09-30 11:41:24.206235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:41:24.206251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:44:51.239106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:44:51.239156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:44:51.239222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:41:25.616064 (xid=0x830x) >[2012-09-30 11:44:51.239243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:44:51.239260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:48:18.270114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:48:18.270170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:48:18.270244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:44:52.647750 (xid=0x831x) >[2012-09-30 11:48:18.270265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:48:18.270282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:51:09.705351] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:51:09.708414] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:51:09.710779] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 11:51:45.304115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:51:45.304157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:51:45.304221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:48:19.681082 (xid=0x832x) >[2012-09-30 11:51:45.304242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:51:45.304258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:55:12.334100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:55:12.334148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:55:12.334216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:51:46.711576 (xid=0x833x) >[2012-09-30 11:55:12.334237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:55:12.334254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 11:58:39.363120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 11:58:39.363168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 11:58:39.363235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:55:13.740045 (xid=0x834x) >[2012-09-30 11:58:39.363256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 11:58:39.363272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:01:09.792470] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:01:09.795525] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:01:09.797892] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:02:06.393117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:02:06.393164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:02:06.393227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 11:58:40.770629 (xid=0x835x) >[2012-09-30 12:02:06.393248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:02:06.393263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:05:33.425106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:05:33.425156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:05:33.425222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:02:07.802283 (xid=0x836x) >[2012-09-30 12:05:33.425244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:05:33.425260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:09:00.455119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:09:00.455166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:09:00.455232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:05:34.832410 (xid=0x837x) >[2012-09-30 12:09:00.455253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:09:00.455269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:11:09.883941] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:11:09.887013] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:11:09.889458] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:12:27.485119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:12:27.485167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:12:27.485233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:09:01.862525 (xid=0x838x) >[2012-09-30 12:12:27.485254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:12:27.485271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:15:54.519120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:15:54.519170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:15:54.519236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:12:28.896408 (xid=0x839x) >[2012-09-30 12:15:54.519257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:15:54.519274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:19:21.549102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:19:21.549157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:19:21.549230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:15:55.926404 (xid=0x840x) >[2012-09-30 12:19:21.549253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:19:21.549270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:21:09.974688] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:21:09.977180] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:21:09.979333] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:22:48.580090] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:22:48.580136] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:22:48.580214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:19:22.957059 (xid=0x841x) >[2012-09-30 12:22:48.580234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:22:48.580250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:26:15.613073] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:26:15.613129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:26:15.613204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:22:49.989922 (xid=0x842x) >[2012-09-30 12:26:15.613226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:26:15.613243] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:29:42.644103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:29:42.644154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:29:42.644220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:26:17.020893 (xid=0x843x) >[2012-09-30 12:29:42.644241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:29:42.644258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:31:10.066850] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:31:10.069903] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:31:10.072300] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:33:09.674105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:33:09.674150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:33:09.674214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:29:44.051535 (xid=0x844x) >[2012-09-30 12:33:09.674246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:33:09.674264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:36:36.707100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:36:36.707160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:36:36.707227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:33:11.084267 (xid=0x845x) >[2012-09-30 12:36:36.707249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:36:36.707265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:40:03.738104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:40:03.738151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:40:03.738218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:36:38.115432 (xid=0x846x) >[2012-09-30 12:40:03.738240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:40:03.738257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:41:10.161976] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:41:10.164802] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:41:10.167307] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:43:30.772101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:43:30.772153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:43:30.772219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:40:05.148904 (xid=0x847x) >[2012-09-30 12:43:30.772241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:43:30.772257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:46:57.807104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:46:57.807151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:46:57.807215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:43:32.184073 (xid=0x848x) >[2012-09-30 12:46:57.807236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:46:57.807253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:50:27.840107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:50:27.840155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:50:27.840222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:47:02.217181 (xid=0x849x) >[2012-09-30 12:50:27.840243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:50:27.840260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:51:10.260616] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:51:10.263632] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:51:10.265974] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 12:53:54.874116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:53:54.874171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:53:54.874246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:50:29.250829 (xid=0x850x) >[2012-09-30 12:53:54.875184] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:53:54.875221] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 12:57:21.909091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 12:57:21.909139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 12:57:21.909218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:53:56.286216 (xid=0x851x) >[2012-09-30 12:57:21.909239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 12:57:21.909255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:00:48.942122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:00:48.942172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:00:48.942239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 12:57:23.319634 (xid=0x852x) >[2012-09-30 13:00:48.942260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:00:48.942277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:01:10.360088] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:01:10.363563] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:01:10.366106] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:04:15.974113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:04:15.974173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:04:15.974247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:00:50.351613 (xid=0x853x) >[2012-09-30 13:04:15.974269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:04:15.974286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:07:43.009113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:07:43.009162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:07:43.009229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:04:17.386462 (xid=0x854x) >[2012-09-30 13:07:43.009264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:07:43.009281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:11:10.039100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:11:10.039148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:11:10.039215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:07:44.416395 (xid=0x855x) >[2012-09-30 13:11:10.039236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:11:10.039253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:11:10.447915] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:11:10.451577] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:11:10.454705] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:14:37.072114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:14:37.072165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:14:37.072232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:11:11.449669 (xid=0x856x) >[2012-09-30 13:14:37.072254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:14:37.072270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:18:04.103143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:18:04.103190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:18:04.103255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:14:38.480297 (xid=0x857x) >[2012-09-30 13:18:04.103277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:18:04.103293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:21:10.537577] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:21:10.541071] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:21:10.543286] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:21:31.133149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:21:31.133193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:21:31.133257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:18:05.509895 (xid=0x858x) >[2012-09-30 13:21:31.133277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:21:31.133292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:24:58.165146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:24:58.165195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:24:58.165270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:21:32.542027 (xid=0x859x) >[2012-09-30 13:24:58.166066] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:24:58.166102] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:28:25.195134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:28:25.195182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:28:25.195249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:24:59.572100 (xid=0x860x) >[2012-09-30 13:28:25.195270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:28:25.195287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:31:10.629303] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:31:10.632838] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:31:10.635293] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:31:52.226111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:31:52.226155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:31:52.226217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:28:26.603055 (xid=0x861x) >[2012-09-30 13:31:52.226238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:31:52.226254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:35:19.260148] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:35:19.260207] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:35:19.260281] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:31:53.636878 (xid=0x862x) >[2012-09-30 13:35:19.260303] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:35:19.260320] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:38:46.289154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:38:46.289206] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:38:46.289281] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:35:20.666788 (xid=0x863x) >[2012-09-30 13:38:46.290234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:38:46.290276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:41:10.716178] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:41:10.719424] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:41:10.721603] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:42:13.318111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:42:13.318162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:42:13.318229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:38:47.695185 (xid=0x864x) >[2012-09-30 13:42:13.318250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:42:13.318266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:45:40.349146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:45:40.349195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:45:40.349262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:42:14.726043 (xid=0x865x) >[2012-09-30 13:45:40.349283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:45:40.349300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:49:07.377127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:49:07.377178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:49:07.377244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:45:41.754325 (xid=0x866x) >[2012-09-30 13:49:07.377266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:49:07.377283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:51:10.803572] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:51:10.806435] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:51:10.808677] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 13:52:34.406125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:52:34.406173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:52:34.406253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:49:08.783248 (xid=0x867x) >[2012-09-30 13:52:34.406274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:52:34.406289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:56:01.439111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:56:01.439163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:56:01.439229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:52:35.816075 (xid=0x868x) >[2012-09-30 13:56:01.439251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:56:01.439268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 13:59:28.468104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 13:59:28.468154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 13:59:28.468219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:56:02.845286 (xid=0x869x) >[2012-09-30 13:59:28.468241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 13:59:28.468258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:01:10.891754] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:01:10.895292] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:01:10.898174] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:02:55.497112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:02:55.497159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:02:55.497226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 13:59:29.874371 (xid=0x870x) >[2012-09-30 14:02:55.497258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:02:55.497275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:06:22.527114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:06:22.527171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:06:22.527242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:02:56.904500 (xid=0x871x) >[2012-09-30 14:06:22.527264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:06:22.527281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:09:49.557123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:09:49.557169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:09:49.557233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:06:23.934426 (xid=0x872x) >[2012-09-30 14:09:49.557255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:09:49.557271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:11:10.978876] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:11:10.981742] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:11:10.983900] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:13:16.587140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:13:16.587189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:13:16.587256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:09:50.964334 (xid=0x873x) >[2012-09-30 14:13:16.587277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:13:16.587294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:16:43.619107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:16:43.619159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:16:43.619225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:13:17.996046 (xid=0x874x) >[2012-09-30 14:16:43.619247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:16:43.619263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:20:10.649143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:20:10.649191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:20:10.649257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:16:45.026667 (xid=0x875x) >[2012-09-30 14:20:10.649279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:20:10.649295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:21:11.070691] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:21:11.073349] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:21:11.075370] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:23:37.680084] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:23:37.680135] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:23:37.680203] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:20:12.057610 (xid=0x876x) >[2012-09-30 14:23:37.680224] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:23:37.680241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:27:04.713101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:27:04.713148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:27:04.713225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:23:39.090386 (xid=0x877x) >[2012-09-30 14:27:04.713246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:27:04.713262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:30:31.744120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:30:31.744168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:30:31.744235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:27:06.121791 (xid=0x878x) >[2012-09-30 14:30:31.744256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:30:31.744273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:31:11.163801] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:31:11.167187] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:31:11.169781] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:33:58.776105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:33:58.776158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:33:58.776234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:30:33.153802 (xid=0x879x) >[2012-09-30 14:33:58.777507] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:33:58.777554] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:37:25.810140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:37:25.810196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:37:25.810270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:34:00.187738 (xid=0x880x) >[2012-09-30 14:37:25.810305] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:37:25.810324] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:40:52.842139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:40:52.842188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:40:52.842256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:37:27.219519 (xid=0x881x) >[2012-09-30 14:40:52.842277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:40:52.842294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:41:11.257960] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:41:11.260884] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:41:11.263255] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:44:19.874151] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:44:19.874200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:44:19.874266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:40:54.251396 (xid=0x882x) >[2012-09-30 14:44:19.874288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:44:19.874305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:47:46.907098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:47:46.907148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:47:46.907214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:44:21.284016 (xid=0x883x) >[2012-09-30 14:47:46.907236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:47:46.907253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:51:11.347296] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:51:11.351119] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:51:11.354457] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 14:51:13.939096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:51:13.939137] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:51:13.939202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:47:48.316011 (xid=0x884x) >[2012-09-30 14:51:13.939223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:51:13.939239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:54:40.971093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:54:40.971141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:54:40.971211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:51:15.348040 (xid=0x885x) >[2012-09-30 14:54:40.971232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:54:40.971249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 14:58:08.001116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 14:58:08.001164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 14:58:08.001231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:54:42.378743 (xid=0x886x) >[2012-09-30 14:58:08.001252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 14:58:08.001270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:01:11.435662] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:01:11.438605] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:01:11.440885] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:01:35.031119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:01:35.031161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:01:35.031225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 14:58:09.408069 (xid=0x887x) >[2012-09-30 15:01:35.031245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:01:35.031261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:05:02.062105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:05:02.062157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:05:02.062224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:01:36.439046 (xid=0x888x) >[2012-09-30 15:05:02.062245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:05:02.062261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:08:29.092119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:08:29.092176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:08:29.092250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:05:03.468845 (xid=0x889x) >[2012-09-30 15:08:29.092272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:08:29.092289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:11:11.522800] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:11:11.525645] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:11:11.527900] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:11:56.121124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:11:56.121171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:11:56.121234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:08:30.498634 (xid=0x890x) >[2012-09-30 15:11:56.121255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:11:56.121271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:15:23.153117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:15:23.153175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:15:23.153254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:11:57.530605 (xid=0x891x) >[2012-09-30 15:15:23.154522] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:15:23.154569] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:18:50.183087] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:18:50.183135] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:18:50.183202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:15:24.560209 (xid=0x892x) >[2012-09-30 15:18:50.183223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:18:50.183241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:21:11.609857] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:21:11.612699] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:21:11.614773] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:22:17.211107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:22:17.211152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:22:17.211227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:18:51.588693 (xid=0x893x) >[2012-09-30 15:22:17.211247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:22:17.211263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:25:44.243099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:25:44.243148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:25:44.243215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:22:18.620693 (xid=0x894x) >[2012-09-30 15:25:44.243237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:25:44.243253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:29:11.272059] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:29:11.272109] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:29:11.272176] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:25:45.649625 (xid=0x895x) >[2012-09-30 15:29:11.273171] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:29:11.273214] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:31:11.696101] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:31:11.698907] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:31:11.701809] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:32:38.300111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:32:38.300159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:32:38.300232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:29:12.677676 (xid=0x896x) >[2012-09-30 15:32:38.301177] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:32:38.301219] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:36:05.331138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:36:05.331188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:36:05.331253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:32:39.708748 (xid=0x897x) >[2012-09-30 15:36:05.331275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:36:05.331293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:39:32.361136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:39:32.361188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:39:32.361263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:36:06.737937 (xid=0x898x) >[2012-09-30 15:39:32.361291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:39:32.361310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:41:11.784862] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:41:11.788390] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:41:11.790713] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:42:59.389993] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:42:59.390038] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:42:59.390103] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:39:33.767399 (xid=0x899x) >[2012-09-30 15:42:59.390124] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:42:59.390151] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:46:26.422102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:46:26.422158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:46:26.422232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:43:00.799706 (xid=0x900x) >[2012-09-30 15:46:26.422254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:46:26.422271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:49:53.452112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:49:53.452159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:49:53.452226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:46:27.829563 (xid=0x901x) >[2012-09-30 15:49:53.452247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:49:53.452264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:51:11.874037] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:51:11.877096] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:51:11.879564] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 15:53:20.482092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:53:20.482140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:53:20.482207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:49:54.859379 (xid=0x902x) >[2012-09-30 15:53:20.482229] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:53:20.482246] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 15:56:47.514121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 15:56:47.514169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 15:56:47.514246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:53:21.891748 (xid=0x903x) >[2012-09-30 15:56:47.514267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 15:56:47.514284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:00:14.545159] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:00:14.545208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:00:14.545274] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 15:56:48.921884 (xid=0x904x) >[2012-09-30 16:00:14.545295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:00:14.545312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:01:11.962531] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:01:11.965108] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:01:11.967167] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:03:41.574105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:03:41.574157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:03:41.574221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:00:15.950968 (xid=0x905x) >[2012-09-30 16:03:41.574242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:03:41.574258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:07:08.605111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:07:08.605158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:07:08.605224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:03:42.982236 (xid=0x906x) >[2012-09-30 16:07:08.605255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:07:08.605273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:10:35.635091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:10:35.635141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:10:35.635208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:07:10.012264 (xid=0x907x) >[2012-09-30 16:10:35.635230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:10:35.635247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:11:12.052494] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:11:12.055544] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:11:12.057557] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:14:02.665108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:14:02.665155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:14:02.665222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:10:37.042078 (xid=0x908x) >[2012-09-30 16:14:02.665243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:14:02.665260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:17:29.698134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:17:29.698183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:17:29.698249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:14:04.075631 (xid=0x909x) >[2012-09-30 16:17:29.698271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:17:29.698287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:20:56.729089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:20:56.729142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:20:56.729208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:17:31.106030 (xid=0x910x) >[2012-09-30 16:20:56.729230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:20:56.729246] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:21:12.143186] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:21:12.146645] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:21:12.149237] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:24:23.759054] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:24:23.759111] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:24:23.759185] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:20:58.136223 (xid=0x911x) >[2012-09-30 16:24:23.759206] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:24:23.759224] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:27:50.792148] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:27:50.792196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:27:50.792261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:24:25.169004 (xid=0x912x) >[2012-09-30 16:27:50.792282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:27:50.792298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:31:12.228587] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:31:12.231263] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:31:12.233371] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:31:17.821129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:31:17.821174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:31:17.821235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:27:52.198670 (xid=0x913x) >[2012-09-30 16:31:17.821255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:31:17.821270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:34:44.853106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:34:44.853156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:34:44.853222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:31:19.230658 (xid=0x914x) >[2012-09-30 16:34:44.853244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:34:44.853261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:38:11.883117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:38:11.883167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:38:11.883233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:34:46.260097 (xid=0x915x) >[2012-09-30 16:38:11.883254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:38:11.883271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:41:12.316421] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:41:12.319443] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:41:12.321597] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:41:38.912128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:41:38.912174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:41:38.912238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:38:13.289511 (xid=0x916x) >[2012-09-30 16:41:38.912258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:41:38.912274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:45:05.943152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:45:05.943205] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:45:05.943272] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:41:40.319988 (xid=0x917x) >[2012-09-30 16:45:05.943293] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:45:05.943310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:48:32.971103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:48:32.971156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:48:32.971224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:45:07.348653 (xid=0x918x) >[2012-09-30 16:48:32.971246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:48:32.971263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:51:12.399524] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:51:12.402612] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:51:12.405169] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 16:51:59.999156] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:51:59.999231] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:51:59.999307] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:48:34.376761 (xid=0x919x) >[2012-09-30 16:51:59.999329] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:51:59.999347] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:55:27.035086] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:55:27.035139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:55:27.035206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:52:01.412639 (xid=0x920x) >[2012-09-30 16:55:27.035228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:55:27.035245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 16:58:54.065111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 16:58:54.065160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 16:58:54.065226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:55:28.442639 (xid=0x921x) >[2012-09-30 16:58:54.065247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 16:58:54.065264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:01:12.493137] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:01:12.496263] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:01:12.498731] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:02:21.095140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:02:21.095184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:02:21.095248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 16:58:55.471830 (xid=0x922x) >[2012-09-30 17:02:21.095280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:02:21.095297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:05:48.125117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:05:48.125167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:05:48.125234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:02:22.502204 (xid=0x923x) >[2012-09-30 17:05:48.125256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:05:48.125272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:09:15.153156] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:09:15.153204] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:09:15.153269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:05:49.530018 (xid=0x924x) >[2012-09-30 17:09:15.153291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:09:15.153308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:11:12.576703] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:11:12.580277] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:11:12.582738] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:12:42.181126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:12:42.181174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:12:42.181239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:09:16.558023 (xid=0x925x) >[2012-09-30 17:12:42.181260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:12:42.181288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:16:09.210115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:16:09.210168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:16:09.210235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:12:43.587406 (xid=0x926x) >[2012-09-30 17:16:09.210256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:16:09.210273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:19:36.238133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:19:36.238185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:19:36.238253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:16:10.615763 (xid=0x927x) >[2012-09-30 17:19:36.239212] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:19:36.239250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:21:12.660428] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:21:12.663845] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:21:12.666446] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:23:03.267145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:23:03.267190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:23:03.267256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:19:37.644145 (xid=0x928x) >[2012-09-30 17:23:03.267277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:23:03.267295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:26:30.296112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:26:30.296168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:26:30.296257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:23:04.673174 (xid=0x929x) >[2012-09-30 17:26:30.296279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:26:30.296295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:29:57.324144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:29:57.324195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:29:57.324261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:26:31.701013 (xid=0x930x) >[2012-09-30 17:29:57.324282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:29:57.324299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:31:12.741608] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:31:12.744336] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:31:12.746405] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:33:24.352168] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:33:24.352215] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:33:24.352279] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:29:58.729069 (xid=0x931x) >[2012-09-30 17:33:24.352300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:33:24.352316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:36:51.381066] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:36:51.381115] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:36:51.381181] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:33:25.758040 (xid=0x932x) >[2012-09-30 17:36:51.381214] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:36:51.381231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:40:18.408152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:40:18.408202] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:40:18.408269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:36:52.785474 (xid=0x933x) >[2012-09-30 17:40:18.408290] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:40:18.408307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:41:12.824195] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:41:12.827578] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:41:12.829925] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:43:45.436120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:43:45.436168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:43:45.436235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:40:19.813142 (xid=0x934x) >[2012-09-30 17:43:45.436255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:43:45.436272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:47:12.464116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:47:12.464164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:47:12.464230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:43:46.841613 (xid=0x935x) >[2012-09-30 17:47:12.464252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:47:12.464269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:50:39.492111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:50:39.492161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:50:39.492228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:47:13.869038 (xid=0x936x) >[2012-09-30 17:50:39.492250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:50:39.492267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:51:12.904938] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:51:12.907696] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:51:12.909853] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 17:54:06.519099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:54:06.519149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:54:06.519216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:50:40.896151 (xid=0x937x) >[2012-09-30 17:54:06.519237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:54:06.519254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 17:57:33.550115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 17:57:33.550170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 17:57:33.550244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:54:07.926887 (xid=0x938x) >[2012-09-30 17:57:33.550266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 17:57:33.550284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:01:00.577119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:01:00.577169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:01:00.577256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 17:57:34.954042 (xid=0x939x) >[2012-09-30 18:01:00.578195] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:01:00.578239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:01:12.989112] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:01:12.992561] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:01:12.995047] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:04:27.605116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:04:27.605172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:04:27.605246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:01:01.982554 (xid=0x940x) >[2012-09-30 18:04:27.605268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:04:27.605285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:07:54.636115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:07:54.636165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:07:54.636231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:04:29.013413 (xid=0x941x) >[2012-09-30 18:07:54.636252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:07:54.636269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:11:13.068775] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:11:13.071987] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:11:13.074616] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:11:21.664105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:11:21.664148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:11:21.664213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:07:56.041660 (xid=0x942x) >[2012-09-30 18:11:21.664233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:11:21.664250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:14:48.695114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:14:48.695166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:14:48.695233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:11:23.072388 (xid=0x943x) >[2012-09-30 18:14:48.695255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:14:48.695272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:18:15.723099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:18:15.723148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:18:15.723213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:14:50.100656 (xid=0x944x) >[2012-09-30 18:18:15.723235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:18:15.723252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:21:13.155279] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:21:13.158904] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:21:13.161537] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:21:42.752093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:21:42.752148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:21:42.752211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:18:17.129022 (xid=0x945x) >[2012-09-30 18:21:42.752231] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:21:42.752247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:25:09.783129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:25:09.783177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:25:09.783244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:21:44.160166 (xid=0x946x) >[2012-09-30 18:25:09.783266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:25:09.783282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:28:36.811117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:28:36.811175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:28:36.811258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:25:11.188725 (xid=0x947x) >[2012-09-30 18:28:36.812212] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:28:36.812256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:31:13.241468] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:31:13.244697] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:31:13.246789] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:32:03.840097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:32:03.840142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:32:03.840207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:28:38.217740 (xid=0x948x) >[2012-09-30 18:32:03.840238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:32:03.840255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:35:30.872124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:35:30.872182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:35:30.872249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:32:05.249075 (xid=0x949x) >[2012-09-30 18:35:30.872271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:35:30.872287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:38:57.901099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:38:57.901150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:38:57.901217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:35:32.278030 (xid=0x950x) >[2012-09-30 18:38:57.901238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:38:57.901255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:41:13.327898] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:41:13.330750] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:41:13.333163] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:42:24.930114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:42:24.930162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:42:24.930229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:38:59.307075 (xid=0x951x) >[2012-09-30 18:42:24.930250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:42:24.930276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:45:51.962092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:45:51.962140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:45:51.962207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:42:26.339168 (xid=0x952x) >[2012-09-30 18:45:51.962228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:45:51.962245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:49:18.993014] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:49:18.993061] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:49:18.993126] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:45:53.370095 (xid=0x953x) >[2012-09-30 18:49:18.993147] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:49:18.993164] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:51:13.419986] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:51:13.422421] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:51:13.424441] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 18:52:46.024132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:52:46.024181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:52:46.024244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:49:20.400892 (xid=0x954x) >[2012-09-30 18:52:46.024266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:52:46.024282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:56:13.056103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:56:13.056160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:56:13.056946] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:52:47.433409 (xid=0x955x) >[2012-09-30 18:56:13.056972] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:56:13.056989] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 18:59:40.087129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 18:59:40.087177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 18:59:40.087244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:56:14.464519 (xid=0x956x) >[2012-09-30 18:59:40.087266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 18:59:40.087283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:01:13.512462] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:01:13.515597] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:01:13.517776] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:03:07.119064] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:03:07.119113] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:03:07.119184] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 18:59:41.495832 (xid=0x957x) >[2012-09-30 19:03:07.120147] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:03:07.120196] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:06:34.151122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:06:34.151169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:06:34.151236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:03:08.528421 (xid=0x958x) >[2012-09-30 19:06:34.151269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:06:34.151287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:10:01.182113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:10:01.182162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:10:01.182229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:06:35.559279 (xid=0x959x) >[2012-09-30 19:10:01.182251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:10:01.182268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:11:13.604160] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:11:13.606974] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:11:13.609134] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:13:28.213111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:13:28.213160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:13:28.213226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:10:02.590502 (xid=0x960x) >[2012-09-30 19:13:28.213247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:13:28.213267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:16:55.247107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:16:55.247154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:16:55.247219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:13:29.624621 (xid=0x961x) >[2012-09-30 19:16:55.247240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:16:55.247267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:20:22.279116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:20:22.279166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:20:22.279233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:16:56.656736 (xid=0x962x) >[2012-09-30 19:20:22.279254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:20:22.279271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:21:13.697460] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:21:13.700516] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:21:13.702627] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:23:49.309136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:23:49.309186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:23:49.309251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:20:23.686368 (xid=0x963x) >[2012-09-30 19:23:49.309272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:23:49.309288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:27:16.342118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:27:16.342167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:27:16.342233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:23:50.719267 (xid=0x964x) >[2012-09-30 19:27:16.342255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:27:16.342272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:30:43.372074] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:30:43.372127] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:30:43.372204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:27:17.749305 (xid=0x965x) >[2012-09-30 19:30:43.372226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:30:43.372242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:31:13.790099] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:31:13.793521] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:31:13.796074] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:34:10.403124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:34:10.403173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:34:10.403239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:30:44.780104 (xid=0x966x) >[2012-09-30 19:34:10.403261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:34:10.403277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:37:37.437117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:37:37.437167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:37:37.437244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:34:11.814636 (xid=0x967x) >[2012-09-30 19:37:37.438255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:37:37.438299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:41:04.469136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:41:04.469183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:41:04.469250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:37:38.846216 (xid=0x968x) >[2012-09-30 19:41:04.469282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:41:04.469299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:41:13.883282] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:41:13.885936] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:41:13.888004] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:44:31.501127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:44:31.501182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:44:31.501254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:41:05.878032 (xid=0x969x) >[2012-09-30 19:44:31.501275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:44:31.501292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:47:58.534140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:47:58.534187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:47:58.534253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:44:32.911294 (xid=0x970x) >[2012-09-30 19:47:58.534274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:47:58.534291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:51:13.972754] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:51:13.975527] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:51:13.977671] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 19:51:25.566115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:51:25.566170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:51:25.566236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:47:59.943053 (xid=0x971x) >[2012-09-30 19:51:25.566256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:51:25.566271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:54:52.599158] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:54:52.599211] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:54:52.599281] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:51:26.976519 (xid=0x972x) >[2012-09-30 19:54:52.599310] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:54:52.599329] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 19:58:19.631102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 19:58:19.631150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 19:58:19.631218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:54:54.008504 (xid=0x973x) >[2012-09-30 19:58:19.631239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 19:58:19.631256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:01:14.071938] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:01:14.075285] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:01:14.078181] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:01:46.665087] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:01:46.665142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:01:46.665213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 19:58:21.042186 (xid=0x974x) >[2012-09-30 20:01:46.665243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:01:46.665260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:05:13.700103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:05:13.700152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:05:13.700220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:01:48.077069 (xid=0x975x) >[2012-09-30 20:05:13.700241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:05:13.700258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:08:40.733112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:08:40.733161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:08:40.733227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:05:15.110505 (xid=0x976x) >[2012-09-30 20:08:40.733249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:08:40.733266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:11:14.169690] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:11:14.172502] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:11:14.174875] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:12:07.767146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:12:07.767193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:12:07.767259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:08:42.143817 (xid=0x977x) >[2012-09-30 20:12:07.767289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:12:07.767307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:15:34.801110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:15:34.801159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:15:34.801227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:12:09.178385 (xid=0x978x) >[2012-09-30 20:15:34.801249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:15:34.801266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:19:01.833126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:19:01.833178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:19:01.833251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:15:36.210415 (xid=0x979x) >[2012-09-30 20:19:01.834058] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:19:01.834102] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:21:14.265790] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:21:14.268482] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:21:14.270571] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:22:28.866135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:22:28.866179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:22:28.866242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:19:03.243429 (xid=0x980x) >[2012-09-30 20:22:28.866263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:22:28.866279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:25:55.900136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:25:55.900194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:25:55.900261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:22:30.277294 (xid=0x981x) >[2012-09-30 20:25:55.900282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:25:55.900298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:29:22.932146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:29:22.932198] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:29:22.932264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:25:57.309646 (xid=0x982x) >[2012-09-30 20:29:22.932286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:29:22.932302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:31:14.358105] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:31:14.360990] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:31:14.363788] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:32:49.963123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:32:49.963170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:32:49.963236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:29:24.340399 (xid=0x983x) >[2012-09-30 20:32:49.963257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:32:49.963273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:36:16.994122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:36:16.994172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:36:16.994237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:32:51.371575 (xid=0x984x) >[2012-09-30 20:36:16.994268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:36:16.994286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:39:44.024109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:39:44.024158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:39:44.024223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:36:18.401797 (xid=0x985x) >[2012-09-30 20:39:44.024245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:39:44.024262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:41:14.445173] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:41:14.448052] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:41:14.450503] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:43:11.053115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:43:11.053163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:43:11.053229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:39:45.430680 (xid=0x986x) >[2012-09-30 20:43:11.053251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:43:11.053267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:46:38.082118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:46:38.082170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:46:38.082243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:43:12.459170 (xid=0x987x) >[2012-09-30 20:46:38.082266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:46:38.082295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:50:05.109137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:50:05.109185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:50:05.109251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:46:39.486681 (xid=0x988x) >[2012-09-30 20:50:05.109272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:50:05.109289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:51:14.526216] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:51:14.529694] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:51:14.532433] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 20:53:32.138123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:53:32.138181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:53:32.138254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:50:06.513841 (xid=0x989x) >[2012-09-30 20:53:32.138276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:53:32.138293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 20:56:59.166145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 20:56:59.166217] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 20:56:59.166298] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:53:33.542889 (xid=0x990x) >[2012-09-30 20:56:59.166321] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 20:56:59.166339] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:00:26.195121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:00:26.195171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:00:26.195248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 20:57:00.572071 (xid=0x991x) >[2012-09-30 21:00:26.195269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:00:26.195285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:01:14.611006] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:01:14.613740] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:01:14.615980] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:03:53.224073] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:03:53.224122] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:03:53.224189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:00:27.601281 (xid=0x992x) >[2012-09-30 21:03:53.224211] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:03:53.224228] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:07:20.254100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:07:20.254147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:07:20.254211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:03:54.631014 (xid=0x993x) >[2012-09-30 21:07:20.254233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:07:20.254250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:10:47.282104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:10:47.282154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:10:47.282218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:07:21.659500 (xid=0x994x) >[2012-09-30 21:10:47.282252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:10:47.282269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:11:14.694334] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:11:14.697112] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:11:14.699541] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:14:14.310114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:14:14.310162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:14:14.310228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:10:48.687388 (xid=0x995x) >[2012-09-30 21:14:14.310249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:14:14.310266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:17:41.339060] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:17:41.339115] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:17:41.339189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:14:15.716507 (xid=0x996x) >[2012-09-30 21:17:41.339212] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:17:41.339230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:21:08.368073] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:21:08.368122] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:21:08.368188] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:17:42.744860 (xid=0x997x) >[2012-09-30 21:21:08.368209] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:21:08.368236] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:21:14.777788] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:21:14.780406] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:21:14.782546] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:24:35.396118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:24:35.396168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:24:35.396235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:21:09.773280 (xid=0x998x) >[2012-09-30 21:24:35.396257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:24:35.396273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:28:02.427119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:28:02.427168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:28:02.427234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:24:36.804216 (xid=0x999x) >[2012-09-30 21:28:02.427255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:28:02.427272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:31:14.864250] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:31:14.867394] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:31:14.869760] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:31:29.458112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:31:29.458160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:31:29.458228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:28:03.834853 (xid=0x1000x) >[2012-09-30 21:31:29.458259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:31:29.458276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:34:56.490123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:34:56.490177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:34:56.490243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:31:30.867216 (xid=0x1001x) >[2012-09-30 21:34:56.490265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:34:56.490282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:38:23.520105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:38:23.520156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:38:23.520223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:34:57.897668 (xid=0x1002x) >[2012-09-30 21:38:23.520244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:38:23.520261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:41:14.955161] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:41:14.957978] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:41:14.960202] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:41:50.551143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:41:50.551191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:41:50.551257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:38:24.928616 (xid=0x1003x) >[2012-09-30 21:41:50.551288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:41:50.551305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:45:17.585129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:45:17.585177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:45:17.585243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:41:51.962376 (xid=0x1004x) >[2012-09-30 21:45:17.585264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:45:17.585281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:48:44.617120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:48:44.617168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:48:44.617233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:45:18.993877 (xid=0x1005x) >[2012-09-30 21:48:44.617255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:48:44.617272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:51:15.051359] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:51:15.055228] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:51:15.057955] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 21:52:11.649124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:52:11.649171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:52:11.649237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:48:46.026029 (xid=0x1006x) >[2012-09-30 21:52:11.649258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:52:11.649275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:55:38.684099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:55:38.684167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:55:38.684240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:52:13.061650 (xid=0x1007x) >[2012-09-30 21:55:38.684261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:55:38.684278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 21:59:05.711132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 21:59:05.711180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 21:59:05.711246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:55:40.089035 (xid=0x1008x) >[2012-09-30 21:59:05.711268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 21:59:05.711285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:01:15.139765] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:01:15.143403] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:01:15.145846] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:02:32.743145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:02:32.743197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:02:32.743262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 21:59:07.120149 (xid=0x1009x) >[2012-09-30 22:02:32.743282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:02:32.743299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:05:59.773130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:05:59.773179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:05:59.773246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:02:34.150842 (xid=0x1010x) >[2012-09-30 22:05:59.773278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:05:59.773295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:09:26.805111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:09:26.805161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:09:26.805225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:06:01.182715 (xid=0x1011x) >[2012-09-30 22:09:26.805246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:09:26.805263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:11:15.230134] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:11:15.233461] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:11:15.235833] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:12:53.835107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:12:53.835159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:12:53.835226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:09:28.211919 (xid=0x1012x) >[2012-09-30 22:12:53.835247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:12:53.835265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:16:20.866107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:16:20.866153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:16:20.866218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:12:55.243084 (xid=0x1013x) >[2012-09-30 22:16:20.866251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:16:20.866268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:19:47.894128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:19:47.894180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:19:47.894246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:16:22.271353 (xid=0x1014x) >[2012-09-30 22:19:47.894267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:19:47.894284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:21:15.314880] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:21:15.318168] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:21:15.320738] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:23:14.923128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:23:14.923178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:23:14.923245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:19:49.299887 (xid=0x1015x) >[2012-09-30 22:23:14.923266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:23:14.923282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:26:41.953131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:26:41.953181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:26:41.953250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:23:16.330744 (xid=0x1016x) >[2012-09-30 22:26:41.953278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:26:41.953297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:30:08.981129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:30:08.981187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:30:08.981254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:26:43.358265 (xid=0x1017x) >[2012-09-30 22:30:08.981275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:30:08.981291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:31:15.398159] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:31:15.400945] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:31:15.402957] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:33:36.009133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:33:36.009190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:33:36.009262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:30:10.386558 (xid=0x1018x) >[2012-09-30 22:33:36.009283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:33:36.009300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:37:03.039141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:37:03.039189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:37:03.039252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:33:37.416269 (xid=0x1019x) >[2012-09-30 22:37:03.039273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:37:03.039290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:40:30.067106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:40:30.067154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:40:30.067220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:37:04.444336 (xid=0x1020x) >[2012-09-30 22:40:30.067251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:40:30.067269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:41:15.482237] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:41:15.484804] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:41:15.487192] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:43:57.095112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:43:57.095162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:43:57.095229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:40:31.472699 (xid=0x1021x) >[2012-09-30 22:43:57.095251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:43:57.095268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:47:24.126108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:47:24.126155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:47:24.126220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:43:58.503404 (xid=0x1022x) >[2012-09-30 22:47:24.126241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:47:24.126258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:50:51.152111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:50:51.152165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:50:51.152230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:47:25.529558 (xid=0x1023x) >[2012-09-30 22:50:51.152252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:50:51.152281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:51:15.565115] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:51:15.568798] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:51:15.571475] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 22:54:18.179103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:54:18.179154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:54:18.179219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:50:52.556372 (xid=0x1024x) >[2012-09-30 22:54:18.179240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:54:18.179257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 22:57:45.211120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 22:57:45.211176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 22:57:45.211248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:54:19.587838 (xid=0x1025x) >[2012-09-30 22:57:45.211270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 22:57:45.211287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:01:12.238128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:01:12.238176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:01:12.238243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 22:57:46.615262 (xid=0x1026x) >[2012-09-30 23:01:12.238264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:01:12.238281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:01:15.649705] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:01:15.652898] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:01:15.655125] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:04:39.268130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:04:39.268188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:04:39.268263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:01:13.645078 (xid=0x1027x) >[2012-09-30 23:04:39.268284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:04:39.268302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:08:06.299154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:08:06.299204] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:08:06.299271] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:04:40.676426 (xid=0x1028x) >[2012-09-30 23:08:06.299292] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:08:06.299309] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:11:15.731092] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:11:15.733835] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:11:15.736151] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:11:33.325119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:11:33.325167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:11:33.325231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:08:07.702951 (xid=0x1029x) >[2012-09-30 23:11:33.325263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:11:33.325279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:15:00.357070] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:15:00.357119] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:15:00.357186] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:11:34.734524 (xid=0x1030x) >[2012-09-30 23:15:00.357208] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:15:00.357225] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:18:27.385104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:18:27.385152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:18:27.385219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:15:01.762326 (xid=0x1031x) >[2012-09-30 23:18:27.385240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:18:27.385257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:21:15.819394] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:21:15.823025] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:21:15.825430] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:21:54.416124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:21:54.416174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:21:54.416240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:18:28.793158 (xid=0x1032x) >[2012-09-30 23:21:54.416260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:21:54.416276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:25:21.448094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:25:21.448143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:25:21.448208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:21:55.825440 (xid=0x1033x) >[2012-09-30 23:25:21.448229] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:25:21.448246] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:28:48.475126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:28:48.475174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:28:48.475242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:25:22.852885 (xid=0x1034x) >[2012-09-30 23:28:48.475263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:28:48.475280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:31:15.905234] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:31:15.908459] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:31:15.911118] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:32:15.506110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:32:15.506156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:32:15.506221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:28:49.883378 (xid=0x1035x) >[2012-09-30 23:32:15.506243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:32:15.506259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:35:42.536104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:35:42.536160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:35:42.536246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:32:16.913315 (xid=0x1036x) >[2012-09-30 23:35:42.536268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:35:42.536285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:39:09.565115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:39:09.565165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:39:09.565230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:35:43.942077 (xid=0x1037x) >[2012-09-30 23:39:09.565252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:39:09.565269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:41:15.991973] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:41:15.995024] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:41:15.997391] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:42:36.593143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:42:36.593192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:42:36.593259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:39:10.970722 (xid=0x1038x) >[2012-09-30 23:42:36.593280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:42:36.593297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:46:03.625127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:46:03.625175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:46:03.625241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:42:38.002540 (xid=0x1039x) >[2012-09-30 23:46:03.625272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:46:03.625290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:49:30.654125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:49:30.654177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:49:30.654247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:46:05.030879 (xid=0x1040x) >[2012-09-30 23:49:30.655135] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:49:30.655172] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:51:16.075756] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:51:16.078285] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:51:16.080399] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-09-30 23:52:57.682101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:52:57.682150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:52:57.682215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:49:32.059235 (xid=0x1041x) >[2012-09-30 23:52:57.682237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:52:57.682253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:56:24.713097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:56:24.713146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:56:24.713213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:52:59.090619 (xid=0x1042x) >[2012-09-30 23:56:24.713234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:56:24.713251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-09-30 23:59:51.742097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-09-30 23:59:51.742154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-09-30 23:59:51.742222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:56:26.119353 (xid=0x1043x) >[2012-09-30 23:59:51.742242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-09-30 23:59:51.742259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:01:16.162626] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:01:16.166633] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:01:16.169581] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:03:18.771121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:03:18.771171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:03:18.771238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-09-30 23:59:53.148513 (xid=0x1044x) >[2012-10-01 00:03:18.771259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:03:18.771276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:06:45.801140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:06:45.801189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:06:45.801256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:03:20.178433 (xid=0x1045x) >[2012-10-01 00:06:45.801277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:06:45.801295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:10:12.834146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:10:12.834198] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:10:12.834267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:06:47.211377 (xid=0x1046x) >[2012-10-01 00:10:12.834298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:10:12.834316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:11:16.257021] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:11:16.260439] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:11:16.262897] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:13:39.864143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:13:39.864193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:13:39.864259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:10:14.241162 (xid=0x1047x) >[2012-10-01 00:13:39.864281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:13:39.864297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:17:06.905129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:17:06.905178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:17:06.905243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:13:41.282363 (xid=0x1048x) >[2012-10-01 00:17:06.905264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:17:06.905281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:20:33.932117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:20:33.932168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:20:33.932235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:17:08.309422 (xid=0x1049x) >[2012-10-01 00:20:33.932268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:20:33.932285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:21:16.345772] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:21:16.349424] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:21:16.352496] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:24:00.959127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:24:00.959182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:24:00.959248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:20:35.336315 (xid=0x1050x) >[2012-10-01 00:24:00.959269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:24:00.959285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:27:27.988137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:27:27.988188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:27:27.988252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:24:02.365543 (xid=0x1051x) >[2012-10-01 00:27:27.988273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:27:27.988289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:30:55.015099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:30:55.015150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:30:55.015216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:27:29.392924 (xid=0x1052x) >[2012-10-01 00:30:55.015237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:30:55.015255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:31:16.428375] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:31:16.430924] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:31:16.433199] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:34:22.043104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:34:22.043154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:34:22.043221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:30:56.420301 (xid=0x1053x) >[2012-10-01 00:34:22.043242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:34:22.043258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:37:49.075139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:37:49.075194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:37:49.075266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:34:23.452565 (xid=0x1054x) >[2012-10-01 00:37:49.075288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:37:49.075305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:41:16.102141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:41:16.102190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:41:16.102254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:37:50.479998 (xid=0x1055x) >[2012-10-01 00:41:16.102275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:41:16.102291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:41:16.508077] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:41:16.511058] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:41:16.513367] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:44:43.132153] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:44:43.132211] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:44:43.132284] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:41:17.510010 (xid=0x1056x) >[2012-10-01 00:44:43.132306] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:44:43.132324] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:48:10.161135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:48:10.161182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:48:10.161250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:44:44.538083 (xid=0x1057x) >[2012-10-01 00:48:10.161271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:48:10.161288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:51:16.590587] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:51:16.594435] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:51:16.597219] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 00:51:37.188155] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:51:37.188200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:51:37.188261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:48:11.564989 (xid=0x1058x) >[2012-10-01 00:51:37.188281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:51:37.188297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:55:04.218133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:55:04.218182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:55:04.218248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:51:38.595089 (xid=0x1059x) >[2012-10-01 00:55:04.218269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:55:04.218286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 00:58:31.245111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 00:58:31.245163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 00:58:31.245229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:55:05.622335 (xid=0x1060x) >[2012-10-01 00:58:31.245250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 00:58:31.245267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:01:16.674896] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:01:16.677954] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:01:16.680465] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:01:58.274145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:01:58.274190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:01:58.274252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 00:58:32.651562 (xid=0x1061x) >[2012-10-01 01:01:58.274272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:01:58.274288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:05:25.304140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:05:25.304192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:05:25.304269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:01:59.681661 (xid=0x1062x) >[2012-10-01 01:05:25.304291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:05:25.304307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:08:52.334144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:08:52.334194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:08:52.334261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:05:26.711087 (xid=0x1063x) >[2012-10-01 01:08:52.334283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:08:52.334299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:11:16.761452] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:11:16.763897] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:11:16.766088] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:12:19.362083] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:12:19.362129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:12:19.362194] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:08:53.739852 (xid=0x1064x) >[2012-10-01 01:12:19.362215] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:12:19.362231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:15:46.393113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:15:46.393164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:15:46.393231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:12:20.770708 (xid=0x1065x) >[2012-10-01 01:15:46.393263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:15:46.393280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:19:13.422106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:19:13.422156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:19:13.422223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:15:47.799671 (xid=0x1066x) >[2012-10-01 01:19:13.422244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:19:13.422261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:21:16.847597] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:21:16.850377] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:21:16.852616] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:22:40.450085] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:22:40.450133] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:22:40.450198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:19:14.828041 (xid=0x1067x) >[2012-10-01 01:22:40.450219] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:22:40.450235] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:26:07.482114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:26:07.482163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:26:07.482229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:22:41.859113 (xid=0x1068x) >[2012-10-01 01:26:07.482250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:26:07.482267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:29:34.512116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:29:34.512167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:29:34.512233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:26:08.889127 (xid=0x1069x) >[2012-10-01 01:29:34.512254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:29:34.512271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:31:16.934145] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:31:16.937638] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:31:16.940024] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:33:01.539105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:33:01.539153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:33:01.539220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:29:35.916376 (xid=0x1070x) >[2012-10-01 01:33:01.539240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:33:01.539257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:36:28.569113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:36:28.569162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:36:28.569227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:33:02.946546 (xid=0x1071x) >[2012-10-01 01:36:28.569249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:36:28.569265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:39:55.600056] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:39:55.600104] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:39:55.600185] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:36:29.976835 (xid=0x1072x) >[2012-10-01 01:39:55.600206] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:39:55.600222] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:41:17.022331] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:41:17.024761] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:41:17.026955] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:43:22.630118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:43:22.630166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:43:22.630233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:39:57.007016 (xid=0x1073x) >[2012-10-01 01:43:22.630254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:43:22.630270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:46:49.661112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:46:49.661159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:46:49.661226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:43:24.038143 (xid=0x1074x) >[2012-10-01 01:46:49.661248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:46:49.661265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:50:16.693106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:50:16.693157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:50:16.693223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:46:51.070129 (xid=0x1075x) >[2012-10-01 01:50:16.693254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:50:16.693272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:51:17.113887] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:51:17.117310] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:51:17.119579] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 01:53:43.724120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:53:43.724168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:53:43.724236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:50:18.101416 (xid=0x1076x) >[2012-10-01 01:53:43.724258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:53:43.724274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 01:57:10.754125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 01:57:10.754174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 01:57:10.754242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:53:45.131646 (xid=0x1077x) >[2012-10-01 01:57:10.754263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 01:57:10.754280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:00:37.783116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:00:37.783166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:00:37.783231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 01:57:12.161042 (xid=0x1078x) >[2012-10-01 02:00:37.783252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:00:37.783269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:01:17.199687] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:01:17.203181] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:01:17.205749] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:04:04.813098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:04:04.813152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:04:04.813219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:00:39.191055 (xid=0x1079x) >[2012-10-01 02:04:04.813240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:04:04.813257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:07:31.843103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:07:31.843153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:07:31.843220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:04:06.220062 (xid=0x1080x) >[2012-10-01 02:07:31.843242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:07:31.843258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:10:58.872106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:10:58.872156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:10:58.872224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:07:33.249102 (xid=0x1081x) >[2012-10-01 02:10:58.872246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:10:58.872263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:11:17.286390] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:11:17.289702] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:11:17.291876] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:14:25.902099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:14:25.902147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:14:25.902212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:11:00.279689 (xid=0x1082x) >[2012-10-01 02:14:25.902234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:14:25.902250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:17:52.933099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:17:52.933149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:17:52.933217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:14:27.311020 (xid=0x1083x) >[2012-10-01 02:17:52.933238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:17:52.933255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:21:17.369596] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:21:17.372106] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:21:17.374383] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:21:19.963097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:21:19.963134] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:21:19.963197] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:17:54.340050 (xid=0x1084x) >[2012-10-01 02:21:19.963218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:21:19.963247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:24:46.994106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:24:46.994165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:24:46.994240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:21:21.370977 (xid=0x1085x) >[2012-10-01 02:24:46.994262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:24:46.994280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:28:17.023124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:28:17.023175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:28:17.023240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:24:51.400160 (xid=0x1086x) >[2012-10-01 02:28:17.023262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:28:17.023279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:31:17.459791] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:31:17.462689] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:31:17.464949] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:31:44.056129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:31:44.056179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:31:44.056243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:28:18.433342 (xid=0x1087x) >[2012-10-01 02:31:44.056265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:31:44.056280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:35:11.088080] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:35:11.088129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:35:11.088210] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:31:45.464867 (xid=0x1088x) >[2012-10-01 02:35:11.088231] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:35:11.088248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:38:38.115141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:38:38.115192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:38:38.115266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:35:12.491896 (xid=0x1089x) >[2012-10-01 02:38:38.116220] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:38:38.116257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:41:17.540946] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:41:17.544373] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:41:17.546981] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:42:05.141108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:42:05.141151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:42:05.141216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:38:39.518750 (xid=0x1090x) >[2012-10-01 02:42:05.141237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:42:05.141253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:45:32.172123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:45:32.172172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:45:32.172238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:42:06.549097 (xid=0x1091x) >[2012-10-01 02:45:32.172272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:45:32.172289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:48:59.199117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:48:59.199167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:48:59.199233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:45:33.576676 (xid=0x1092x) >[2012-10-01 02:48:59.199255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:48:59.199271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:51:17.627631] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:51:17.631038] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:51:17.633634] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 02:52:26.228122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:52:26.228175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:52:26.228242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:49:00.606021 (xid=0x1093x) >[2012-10-01 02:52:26.228263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:52:26.228280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:55:53.260084] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:55:53.260137] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:55:53.260214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:52:27.637349 (xid=0x1094x) >[2012-10-01 02:55:53.260237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:55:53.260267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 02:59:20.288125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 02:59:20.288174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 02:59:20.288241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:55:54.665086 (xid=0x1095x) >[2012-10-01 02:59:20.288263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 02:59:20.288280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:01:17.713107] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:01:17.715904] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:01:17.718075] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:02:47.317125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:02:47.317186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:02:47.317267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 02:59:21.693865 (xid=0x1096x) >[2012-10-01 03:02:47.318069] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:02:47.318112] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:06:14.348100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:06:14.348152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:06:14.348219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:02:48.725714 (xid=0x1097x) >[2012-10-01 03:06:14.348240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:06:14.348257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:09:41.377084] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:09:41.377136] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:09:41.377213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:06:15.754414 (xid=0x1098x) >[2012-10-01 03:09:41.377234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:09:41.377250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:11:17.797665] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:11:17.800786] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:11:17.802830] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:13:08.406107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:13:08.406157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:13:08.406224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:09:42.783187 (xid=0x1099x) >[2012-10-01 03:13:08.406245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:13:08.406261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:16:35.439110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:16:35.439162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:16:35.439229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:13:09.816476 (xid=0x1100x) >[2012-10-01 03:16:35.439251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:16:35.439267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:20:02.465134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:20:02.465185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:20:02.465251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:16:36.842379 (xid=0x1101x) >[2012-10-01 03:20:02.465283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:20:02.465301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:21:17.882908] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:21:17.885744] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:21:17.888285] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:23:29.493113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:23:29.493159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:23:29.493224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:20:03.870697 (xid=0x1102x) >[2012-10-01 03:23:29.493244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:23:29.493261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:26:56.522123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:26:56.522178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:26:56.522253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:23:30.899703 (xid=0x1103x) >[2012-10-01 03:26:56.522276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:26:56.522293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:30:23.550072] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:30:23.550123] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:30:23.550188] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:26:57.927563 (xid=0x1104x) >[2012-10-01 03:30:23.550209] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:30:23.550225] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:31:17.965742] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:31:17.968106] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:31:17.970645] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:33:50.578129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:33:50.578179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:33:50.578245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:30:24.956037 (xid=0x1105x) >[2012-10-01 03:33:50.578266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:33:50.578282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:37:17.607110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:37:17.607158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:37:17.607224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:33:51.984800 (xid=0x1106x) >[2012-10-01 03:37:17.607245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:37:17.607262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:40:44.635111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:40:44.635164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:40:44.635229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:37:19.012767 (xid=0x1107x) >[2012-10-01 03:40:44.635250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:40:44.635267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:41:18.049223] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:41:18.052495] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:41:18.055049] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:44:11.664108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:44:11.664161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:44:11.664234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:40:46.041600 (xid=0x1108x) >[2012-10-01 03:44:11.665179] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:44:11.665223] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:47:38.694101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:47:38.694152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:47:38.694219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:44:13.071252 (xid=0x1109x) >[2012-10-01 03:47:38.694240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:47:38.694257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:51:05.722107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:51:05.722162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:51:05.722231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:47:40.099599 (xid=0x1110x) >[2012-10-01 03:51:05.722252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:51:05.722269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:51:18.141036] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:51:18.144606] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:51:18.147119] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 03:54:32.757113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:54:32.757165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:54:32.757231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:51:07.134501 (xid=0x1111x) >[2012-10-01 03:54:32.757253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:54:32.757269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 03:57:59.788124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 03:57:59.788172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 03:57:59.788239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:54:34.164829 (xid=0x1112x) >[2012-10-01 03:57:59.788261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 03:57:59.788277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:01:18.219043] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:01:18.221614] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:01:18.223892] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:01:26.815118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:01:26.815163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:01:26.815227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 03:58:01.191919 (xid=0x1113x) >[2012-10-01 04:01:26.815248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:01:26.815264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:04:53.846117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:04:53.846183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:04:53.846257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:01:28.223782 (xid=0x1114x) >[2012-10-01 04:04:53.846279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:04:53.846295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:08:20.873104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:08:20.873153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:08:20.873220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:04:55.250588 (xid=0x1115x) >[2012-10-01 04:08:20.873242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:08:20.873258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:11:18.303386] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:11:18.306508] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:11:18.309606] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:11:47.900096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:11:47.900152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:11:47.900228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:08:22.277482 (xid=0x1116x) >[2012-10-01 04:11:47.900249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:11:47.900267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:15:14.933122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:15:14.933172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:15:14.933239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:11:49.310308 (xid=0x1117x) >[2012-10-01 04:15:14.933271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:15:14.933289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:18:41.962109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:18:41.962163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:18:41.962226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:15:16.339168 (xid=0x1118x) >[2012-10-01 04:18:41.962247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:18:41.962264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:21:18.390064] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:21:18.392176] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:21:18.394472] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:22:08.988108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:22:08.988156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:22:08.988221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:18:43.365920 (xid=0x1119x) >[2012-10-01 04:22:08.988242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:22:08.988258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:25:36.019105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:25:36.019155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:25:36.019219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:22:10.396556 (xid=0x1120x) >[2012-10-01 04:25:36.019240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:25:36.019267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:29:03.046102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:29:03.046153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:29:03.046218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:25:37.423920 (xid=0x1121x) >[2012-10-01 04:29:03.046239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:29:03.046256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:31:18.473817] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:31:18.476962] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:31:18.479476] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:32:30.076103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:32:30.076148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:32:30.076212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:29:04.453648 (xid=0x1122x) >[2012-10-01 04:32:30.076233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:32:30.076249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:35:57.106133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:35:57.106180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:35:57.106247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:32:31.484012 (xid=0x1123x) >[2012-10-01 04:35:57.106269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:35:57.106285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:39:24.134151] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:39:24.134202] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:39:24.134281] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:35:58.511179 (xid=0x1124x) >[2012-10-01 04:39:24.134302] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:39:24.134318] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:41:18.561787] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:41:18.565044] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:41:18.567433] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:42:51.167151] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:42:51.167212] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:42:51.167286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:39:25.544391 (xid=0x1125x) >[2012-10-01 04:42:51.167307] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:42:51.167324] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:46:18.197153] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:46:18.197201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:46:18.197267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:42:52.574351 (xid=0x1126x) >[2012-10-01 04:46:18.197289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:46:18.197305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:49:45.224158] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:49:45.224209] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:49:45.224275] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:46:19.601394 (xid=0x1127x) >[2012-10-01 04:49:45.224308] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:49:45.224325] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:51:18.643583] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:51:18.646320] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:51:18.648558] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 04:53:12.251098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:53:12.251149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:53:12.251225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:49:46.629028 (xid=0x1128x) >[2012-10-01 04:53:12.251252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:53:12.251272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 04:56:39.283178] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 04:56:39.283229] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 04:56:39.283297] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:53:13.660660 (xid=0x1129x) >[2012-10-01 04:56:39.283319] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 04:56:39.283336] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:00:06.312144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:00:06.312197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:00:06.312262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 04:56:40.689555 (xid=0x1130x) >[2012-10-01 05:00:06.312283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:00:06.312311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:01:18.728133] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:01:18.731051] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:01:18.733970] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:03:33.339131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:03:33.339181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:03:33.339248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:00:07.716503 (xid=0x1131x) >[2012-10-01 05:03:33.339269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:03:33.339286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:07:00.368165] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:07:00.368221] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:07:00.368292] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:03:34.745150 (xid=0x1132x) >[2012-10-01 05:07:00.368314] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:07:00.368331] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:10:27.395097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:10:27.395148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:10:27.395213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:07:01.772483 (xid=0x1133x) >[2012-10-01 05:10:27.395235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:10:27.395252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:11:18.810242] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:11:18.813181] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:11:18.815392] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:13:54.422086] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:13:54.422135] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:13:54.422202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:10:28.799833 (xid=0x1134x) >[2012-10-01 05:13:54.422223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:13:54.422239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:17:21.453120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:17:21.453168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:17:21.453234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:13:55.830640 (xid=0x1135x) >[2012-10-01 05:17:21.453256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:17:21.453272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:20:48.482126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:20:48.482173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:20:48.482241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:17:22.859035 (xid=0x1136x) >[2012-10-01 05:20:48.482263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:20:48.482280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:21:18.896023] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:21:18.899045] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:21:18.901432] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:24:15.511127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:24:15.511177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:24:15.511243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:20:49.888157 (xid=0x1137x) >[2012-10-01 05:24:15.511264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:24:15.511281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:27:42.541100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:27:42.541150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:27:42.541216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:24:16.918136 (xid=0x1138x) >[2012-10-01 05:27:42.541238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:27:42.541255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:31:09.571095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:31:09.571143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:31:09.571209] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:27:43.948511 (xid=0x1139x) >[2012-10-01 05:31:09.571231] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:31:09.571247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:31:18.983132] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:31:18.986005] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:31:18.988189] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:34:36.600127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:34:36.600188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:34:36.600255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:31:10.977345 (xid=0x1140x) >[2012-10-01 05:34:36.600276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:34:36.600292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:38:03.636117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:38:03.636165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:38:03.636229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:34:38.013580 (xid=0x1141x) >[2012-10-01 05:38:03.636250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:38:03.636267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:41:19.068295] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:41:19.071599] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:41:19.073992] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:41:30.663141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:41:30.663183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:41:30.663248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:38:05.041025 (xid=0x1142x) >[2012-10-01 05:41:30.663269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:41:30.663285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:44:57.695100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:44:57.695158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:44:57.695232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:41:32.072359 (xid=0x1143x) >[2012-10-01 05:44:57.695266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:44:57.695284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:48:24.726012] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:48:24.726061] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:48:24.726128] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:44:59.103177 (xid=0x1144x) >[2012-10-01 05:48:24.726150] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:48:24.726166] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:51:19.158318] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:51:19.161481] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:51:19.163705] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 05:51:51.754112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:51:51.754167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:51:51.754240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:48:26.132054 (xid=0x1145x) >[2012-10-01 05:51:51.754261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:51:51.754279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:55:18.787112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:55:18.787163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:55:18.787227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:51:53.164138 (xid=0x1146x) >[2012-10-01 05:55:18.787258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:55:18.787275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 05:58:45.816098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 05:58:45.816150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 05:58:45.816217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:55:20.193496 (xid=0x1147x) >[2012-10-01 05:58:45.816239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 05:58:45.816256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:01:19.244597] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:01:19.247288] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:01:19.249963] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:02:12.843111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:02:12.843152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:02:12.843216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 05:58:47.221027 (xid=0x1148x) >[2012-10-01 06:02:12.843236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:02:12.843252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:05:39.876157] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:05:39.876206] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:05:39.876280] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:02:14.253598 (xid=0x1149x) >[2012-10-01 06:05:39.877252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:05:39.877295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:09:06.907129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:09:06.907191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:09:06.907259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:05:41.284079 (xid=0x1150x) >[2012-10-01 06:09:06.907280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:09:06.907296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:11:19.332394] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:11:19.334868] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:11:19.337013] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:12:33.933118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:12:33.933169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:12:33.933237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:09:08.310950 (xid=0x1151x) >[2012-10-01 06:12:33.933259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:12:33.933275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:16:00.968132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:16:00.968189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:16:00.968263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:12:35.345719 (xid=0x1152x) >[2012-10-01 06:16:00.968285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:16:00.968303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:19:27.998126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:19:27.998174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:19:27.998237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:16:02.375281 (xid=0x1153x) >[2012-10-01 06:19:27.998268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:19:27.998284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:21:19.418829] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:21:19.421427] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:21:19.423460] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:22:55.024135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:22:55.024190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:22:55.024265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:19:29.401635 (xid=0x1154x) >[2012-10-01 06:22:55.024286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:22:55.024304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:26:22.055150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:26:22.055201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:26:22.055276] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:22:56.432229 (xid=0x1155x) >[2012-10-01 06:26:22.055992] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:26:22.056029] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:29:49.084150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:29:49.084196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:29:49.084262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:26:23.461326 (xid=0x1156x) >[2012-10-01 06:29:49.084283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:29:49.084309] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:31:19.504498] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:31:19.506887] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:31:19.509169] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:33:16.112154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:33:16.112205] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:33:16.112270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:29:50.489192 (xid=0x1157x) >[2012-10-01 06:33:16.112292] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:33:16.112308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:36:43.141129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:36:43.141181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:36:43.141248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:33:17.518228 (xid=0x1158x) >[2012-10-01 06:36:43.141269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:36:43.141286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:40:10.170152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:40:10.170203] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:40:10.170272] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:36:44.547510 (xid=0x1159x) >[2012-10-01 06:40:10.170293] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:40:10.170310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:41:19.585901] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:41:19.588392] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:41:19.590496] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:43:37.197143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:43:37.197194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:43:37.197259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:40:11.574376 (xid=0x1160x) >[2012-10-01 06:43:37.197279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:43:37.197296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:47:04.227167] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:47:04.227213] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:47:04.227279] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:43:38.604026 (xid=0x1161x) >[2012-10-01 06:47:04.227300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:47:04.227316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:50:31.255109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:50:31.255158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:50:31.255224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:47:05.632589 (xid=0x1162x) >[2012-10-01 06:50:31.255245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:50:31.255262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:51:19.668736] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:51:19.671382] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:51:19.673520] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 06:53:58.282138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:53:58.282191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:53:58.282257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:50:32.659616 (xid=0x1163x) >[2012-10-01 06:53:58.282278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:53:58.282295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 06:57:25.310166] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 06:57:25.310217] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 06:57:25.310284] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:53:59.687573 (xid=0x1164x) >[2012-10-01 06:57:25.310305] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 06:57:25.310322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:00:52.339093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:00:52.339140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:00:52.339204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 06:57:26.716521 (xid=0x1165x) >[2012-10-01 07:00:52.339225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:00:52.339241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:01:19.750500] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:01:19.753524] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:01:19.755762] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:04:19.366101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:04:19.366150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:04:19.366218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:00:53.743343 (xid=0x1166x) >[2012-10-01 07:04:19.366239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:04:19.366255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:07:46.395121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:07:46.395171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:07:46.395238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:04:20.772028 (xid=0x1167x) >[2012-10-01 07:07:46.395259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:07:46.395275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:11:13.423114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:11:13.423164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:11:13.423230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:07:47.800760 (xid=0x1168x) >[2012-10-01 07:11:13.423252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:11:13.423268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:11:19.832271] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:11:19.834674] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:11:19.836788] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:14:40.451129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:14:40.451175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:14:40.451250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:11:14.827912 (xid=0x1169x) >[2012-10-01 07:14:40.451271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:14:40.451287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:18:07.481114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:18:07.481165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:18:07.481231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:14:41.858389 (xid=0x1170x) >[2012-10-01 07:18:07.481253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:18:07.481269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:21:19.912626] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:21:19.915264] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:21:19.917458] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:21:34.510118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:21:34.510162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:21:34.510227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:18:08.887292 (xid=0x1171x) >[2012-10-01 07:21:34.510247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:21:34.510262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:25:01.540125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:25:01.540174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:25:01.540247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:21:35.916872 (xid=0x1172x) >[2012-10-01 07:25:01.540281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:25:01.540299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:28:28.566102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:28:28.566151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:28:28.566215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:25:02.943674 (xid=0x1173x) >[2012-10-01 07:28:28.566237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:28:28.566252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:31:19.995207] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:31:19.998116] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:31:20.000389] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:31:55.595128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:31:55.595173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:31:55.595239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:28:29.972174 (xid=0x1174x) >[2012-10-01 07:31:55.595260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:31:55.595276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:35:22.623104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:35:22.623153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:35:22.623216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:31:57.000313 (xid=0x1175x) >[2012-10-01 07:35:22.623237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:35:22.623253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:38:49.650134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:38:49.650193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:38:49.650261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:35:24.027055 (xid=0x1176x) >[2012-10-01 07:38:49.650282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:38:49.650298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:41:20.385667] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:41:20.388372] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:41:20.390700] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:42:16.988124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:42:16.988167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:42:16.988232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:38:51.365021 (xid=0x1177x) >[2012-10-01 07:42:16.988253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:42:16.988269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:45:44.016137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:45:44.016190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:45:44.016255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:42:18.393712 (xid=0x1178x) >[2012-10-01 07:45:44.016277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:45:44.016293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:49:11.044128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:49:11.044176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:49:11.044244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:45:45.421422 (xid=0x1179x) >[2012-10-01 07:49:11.044277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:49:11.044294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:51:20.470791] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:51:20.473771] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:51:20.475951] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 07:52:38.074134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:52:38.074180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:52:38.074245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:49:12.451217 (xid=0x1180x) >[2012-10-01 07:52:38.074266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:52:38.074282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:56:05.104149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:56:05.104202] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:56:05.104277] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:52:39.481403 (xid=0x1181x) >[2012-10-01 07:56:05.104299] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:56:05.104316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 07:59:32.133132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 07:59:32.133181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 07:59:32.133248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:56:06.510405 (xid=0x1182x) >[2012-10-01 07:59:32.133280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 07:59:32.133298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:01:20.557342] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:01:20.560070] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:01:20.562326] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:02:59.163138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:02:59.163188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:02:59.163256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 07:59:33.539873 (xid=0x1183x) >[2012-10-01 08:02:59.163277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:02:59.163293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:06:26.194141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:06:26.194192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:06:26.194257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:03:00.570892 (xid=0x1184x) >[2012-10-01 08:06:26.194278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:06:26.194294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:09:53.224112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:09:53.224161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:09:53.224227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:06:27.601658 (xid=0x1185x) >[2012-10-01 08:09:53.224248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:09:53.224264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:11:20.648669] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:11:20.651207] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:11:20.653507] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:13:20.256150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:13:20.256193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:13:20.256258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:09:54.632848 (xid=0x1186x) >[2012-10-01 08:13:20.256278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:13:20.256294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:16:47.287168] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:16:47.287219] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:16:47.287283] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:13:21.664343 (xid=0x1187x) >[2012-10-01 08:16:47.287304] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:16:47.287321] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:20:14.316005] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:20:14.316053] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:20:14.316120] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:16:48.693269 (xid=0x1188x) >[2012-10-01 08:20:14.316142] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:20:14.316158] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:21:20.734856] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:21:20.738001] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:21:20.740255] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:23:41.345150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:23:41.345201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:23:41.345267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:20:15.722515 (xid=0x1189x) >[2012-10-01 08:23:41.345288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:23:41.345304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:27:08.375159] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:27:08.375208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:27:08.375279] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:23:42.752731 (xid=0x1190x) >[2012-10-01 08:27:08.375301] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:27:08.375318] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:30:35.405105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:30:35.405156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:30:35.405222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:27:09.782620 (xid=0x1191x) >[2012-10-01 08:30:35.405244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:30:35.405261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:31:20.821556] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:31:20.824635] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:31:20.826918] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:34:02.435136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:34:02.435186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:34:02.435252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:30:36.812067 (xid=0x1192x) >[2012-10-01 08:34:02.435273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:34:02.435289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:37:29.466112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:37:29.466160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:37:29.466222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:34:03.843192 (xid=0x1193x) >[2012-10-01 08:37:29.466243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:37:29.466260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:40:56.496115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:40:56.496166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:40:56.496232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:37:30.872876 (xid=0x1194x) >[2012-10-01 08:40:56.496254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:40:56.496271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:41:20.910375] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:41:20.913216] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:41:20.915911] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:44:23.526103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:44:23.526156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:44:23.526232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:40:57.903180 (xid=0x1195x) >[2012-10-01 08:44:23.526253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:44:23.526269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:47:50.557175] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:47:50.557223] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:47:50.557289] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:44:24.933910 (xid=0x1196x) >[2012-10-01 08:47:50.557310] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:47:50.557327] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:51:17.586127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:51:17.586176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:51:17.586242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:47:51.963497 (xid=0x1197x) >[2012-10-01 08:51:17.586264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:51:17.586280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:51:20.998641] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:51:21.001816] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:51:21.004033] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 08:54:44.617120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:54:44.617168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:54:44.617230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:51:18.994083 (xid=0x1198x) >[2012-10-01 08:54:44.617260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:54:44.617276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 08:58:11.648121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 08:58:11.648173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 08:58:11.648242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:54:46.025779 (xid=0x1199x) >[2012-10-01 08:58:11.648264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 08:58:11.648280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:01:21.085781] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:01:21.088938] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:01:21.091407] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:01:38.679100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:01:38.679146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:01:38.679211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 08:58:13.056530 (xid=0x1200x) >[2012-10-01 09:01:38.679232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:01:38.679248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:05:05.713147] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:05:05.713202] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:05:05.713278] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:01:40.090377 (xid=0x1201x) >[2012-10-01 09:05:05.713299] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:05:05.713317] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:08:32.744170] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:08:32.744219] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:08:32.744286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:05:07.121705 (xid=0x1202x) >[2012-10-01 09:08:32.744307] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:08:32.744323] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:11:21.179469] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:11:21.182377] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:11:21.184588] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:11:59.776157] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:11:59.776213] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:11:59.776288] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:08:34.153201 (xid=0x1203x) >[2012-10-01 09:11:59.776310] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:11:59.776327] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:15:26.810096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:15:26.810147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:15:26.810215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:12:01.187281 (xid=0x1204x) >[2012-10-01 09:15:26.810236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:15:26.810252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:18:53.842102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:18:53.842149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:18:53.842224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:15:28.219022 (xid=0x1205x) >[2012-10-01 09:18:53.842244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:18:53.842259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:21:21.273295] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:21:21.276091] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:21:21.278295] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:22:20.873112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:22:20.873156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:22:20.873220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:18:55.250105 (xid=0x1206x) >[2012-10-01 09:22:20.873241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:22:20.873257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:25:47.906157] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:25:47.906208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:25:47.906273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:22:22.283408 (xid=0x1207x) >[2012-10-01 09:25:47.906294] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:25:47.906311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:29:14.937120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:29:14.937172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:29:14.937238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:25:49.314376 (xid=0x1208x) >[2012-10-01 09:29:14.937269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:29:14.937286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:31:21.364240] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:31:21.367061] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:31:21.369942] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:32:41.967124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:32:41.967169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:32:41.967233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:29:16.344506 (xid=0x1209x) >[2012-10-01 09:32:41.967254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:32:41.967269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:36:08.999138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:36:08.999193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:36:08.999267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:32:43.376406 (xid=0x1210x) >[2012-10-01 09:36:08.999289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:36:08.999307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:39:36.028150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:39:36.028200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:39:36.028264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:36:10.405543 (xid=0x1211x) >[2012-10-01 09:39:36.028285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:39:36.028301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:41:21.452168] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:41:21.454857] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:41:21.456937] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:43:03.057108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:43:03.057166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:43:03.057250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:39:37.434437 (xid=0x1212x) >[2012-10-01 09:43:03.058252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:43:03.058292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:46:30.090150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:46:30.090198] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:46:30.090266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:43:04.467295 (xid=0x1213x) >[2012-10-01 09:46:30.090287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:46:30.090304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:49:57.119130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:49:57.119181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:49:57.119246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:46:31.496334 (xid=0x1214x) >[2012-10-01 09:49:57.119267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:49:57.119284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:51:21.539498] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:51:21.541700] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:51:21.543760] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 09:53:24.148160] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:53:24.148212] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:53:24.148280] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:49:58.525358 (xid=0x1215x) >[2012-10-01 09:53:24.148301] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:53:24.148317] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 09:56:51.180132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 09:56:51.180182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 09:56:51.180248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:53:25.556916 (xid=0x1216x) >[2012-10-01 09:56:51.180269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 09:56:51.180286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:00:18.210120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:00:18.210172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:00:18.210240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 09:56:52.587039 (xid=0x1217x) >[2012-10-01 10:00:18.210261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:00:18.210277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:01:21.629474] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:01:21.632325] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:01:21.634425] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:03:45.240166] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:03:45.240218] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:03:45.240284] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:00:19.617199 (xid=0x1218x) >[2012-10-01 10:03:45.240305] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:03:45.240322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:07:12.274126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:07:12.274175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:07:12.274241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:03:46.650970 (xid=0x1219x) >[2012-10-01 10:07:12.274262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:07:12.274279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:10:39.304160] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:10:39.304208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:10:39.304273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:07:13.680871 (xid=0x1220x) >[2012-10-01 10:10:39.304295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:10:39.304312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:11:21.719614] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:11:21.722070] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:11:21.724217] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:14:06.333159] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:14:06.333217] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:14:06.333306] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:10:40.710542 (xid=0x1221x) >[2012-10-01 10:14:06.333327] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:14:06.333345] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:17:33.365121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:17:33.365170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:17:33.365234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:14:07.742169 (xid=0x1222x) >[2012-10-01 10:17:33.365255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:17:33.365271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:21:00.395110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:21:00.395159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:21:00.395227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:17:34.772021 (xid=0x1223x) >[2012-10-01 10:21:00.395249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:21:00.395266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:21:21.806525] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:21:21.809582] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:21:21.811779] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:24:27.423110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:24:27.423161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:24:27.423227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:21:01.800239 (xid=0x1224x) >[2012-10-01 10:24:27.423259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:24:27.423277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:27:54.454107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:27:54.454158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:27:54.454224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:24:28.830865 (xid=0x1225x) >[2012-10-01 10:27:54.454245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:27:54.454262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:31:21.482126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:31:21.482173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:31:21.482239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:27:55.859178 (xid=0x1226x) >[2012-10-01 10:31:21.482261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:31:21.482278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:31:21.889283] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:31:21.893157] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:31:21.896420] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:34:48.514110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:34:48.514157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:34:48.514223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:31:22.891303 (xid=0x1227x) >[2012-10-01 10:34:48.514244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:34:48.514272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:38:15.543115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:38:15.543169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:38:15.543236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:34:49.920086 (xid=0x1228x) >[2012-10-01 10:38:15.543257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:38:15.543274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:41:21.977558] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:41:21.981405] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:41:21.984401] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:41:42.572096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:41:42.572141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:41:42.572205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:38:16.949249 (xid=0x1229x) >[2012-10-01 10:41:42.572226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:41:42.572242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:45:09.605106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:45:09.605159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:45:09.605231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:41:43.982324 (xid=0x1230x) >[2012-10-01 10:45:09.605253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:45:09.605270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:48:36.637105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:48:36.637153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:48:36.637230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:45:11.014025 (xid=0x1231x) >[2012-10-01 10:48:36.637251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:48:36.637268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:51:22.072020] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:51:22.075328] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:51:22.077887] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 10:52:03.668096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:52:03.668140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:52:03.668202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:48:38.045032 (xid=0x1232x) >[2012-10-01 10:52:03.668223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:52:03.668239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:55:30.702112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:55:30.702162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:55:30.702228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:52:05.079020 (xid=0x1233x) >[2012-10-01 10:55:30.702250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:55:30.702267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 10:58:57.733109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 10:58:57.733157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 10:58:57.733223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:55:32.109911 (xid=0x1234x) >[2012-10-01 10:58:57.733257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 10:58:57.733274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:01:22.164496] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:01:22.168226] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:01:22.171064] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:02:24.764117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:02:24.764168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:02:24.764234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 10:58:59.141396 (xid=0x1235x) >[2012-10-01 11:02:24.764255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:02:24.764271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:05:51.797107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:05:51.797156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:05:51.797223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:02:26.174527 (xid=0x1236x) >[2012-10-01 11:05:51.797244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:05:51.797261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:09:18.828099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:09:18.828148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:09:18.828215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:05:53.205494 (xid=0x1237x) >[2012-10-01 11:09:18.828237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:09:18.828254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:11:22.257664] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:11:22.260615] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:11:22.263165] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:12:45.859101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:12:45.859144] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:12:45.859208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:09:20.236739 (xid=0x1238x) >[2012-10-01 11:12:45.859230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:12:45.859245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:16:12.893110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:16:12.893165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:16:12.893241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:12:47.270650 (xid=0x1239x) >[2012-10-01 11:16:12.894258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:16:12.894300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:19:39.925127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:19:39.925175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:19:39.925242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:16:14.302374 (xid=0x1240x) >[2012-10-01 11:19:39.925263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:19:39.925280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:21:22.351148] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:21:22.354789] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:21:22.358130] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:23:06.956134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:23:06.956183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:23:06.956250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:19:41.333529 (xid=0x1241x) >[2012-10-01 11:23:06.956272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:23:06.956288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:26:33.988097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:26:33.988148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:26:33.988215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:23:08.365037 (xid=0x1242x) >[2012-10-01 11:26:33.988237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:26:33.988254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:30:01.018127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:30:01.018176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:30:01.018243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:26:35.395315 (xid=0x1243x) >[2012-10-01 11:30:01.018266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:30:01.018282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:31:22.439255] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:31:22.442413] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:31:22.445123] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:33:28.047134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:33:28.047185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:33:28.047252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:30:02.424367 (xid=0x1244x) >[2012-10-01 11:33:28.047274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:33:28.047292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:36:55.078124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:36:55.078178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:36:55.078243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:33:29.455409 (xid=0x1245x) >[2012-10-01 11:36:55.078264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:36:55.078281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:40:22.108094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:40:22.108147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:40:22.108212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:36:56.485309 (xid=0x1246x) >[2012-10-01 11:40:22.108234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:40:22.108251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:41:22.527340] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:41:22.531018] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:41:22.534114] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:43:49.137130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:43:49.137195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:43:49.137269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:40:23.514740 (xid=0x1247x) >[2012-10-01 11:43:49.138005] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:43:49.138042] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:47:16.169072] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:47:16.169123] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:47:16.169189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:43:50.546533 (xid=0x1248x) >[2012-10-01 11:47:16.169210] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:47:16.169227] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:50:43.200108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:50:43.200156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:50:43.200220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:47:17.577037 (xid=0x1249x) >[2012-10-01 11:50:43.200241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:50:43.200257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:51:22.617488] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:51:22.621008] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:51:22.623746] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 11:54:10.231109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:54:10.231157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:54:10.231224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:50:44.608138 (xid=0x1250x) >[2012-10-01 11:54:10.231257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:54:10.231275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 11:57:37.264078] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 11:57:37.264132] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 11:57:37.264198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:54:11.640854 (xid=0x1251x) >[2012-10-01 11:57:37.264220] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 11:57:37.264237] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:01:04.294159] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:01:04.294216] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:01:04.294293] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 11:57:38.671040 (xid=0x1252x) >[2012-10-01 12:01:04.294316] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:01:04.294333] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:01:22.705857] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:01:22.709046] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:01:22.711744] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:04:31.322118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:04:31.322169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:04:31.322244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:01:05.699208 (xid=0x1253x) >[2012-10-01 12:04:31.323210] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:04:31.323263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:07:58.351108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:07:58.351157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:07:58.351222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:04:32.728381 (xid=0x1254x) >[2012-10-01 12:07:58.351243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:07:58.351259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:11:22.784556] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:11:22.788080] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:11:22.791151] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:11:25.379116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:11:25.379157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:11:25.379223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:07:59.756626 (xid=0x1255x) >[2012-10-01 12:11:25.379243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:11:25.379259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:14:52.409150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:14:52.409201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:14:52.409267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:11:26.786066 (xid=0x1256x) >[2012-10-01 12:14:52.409289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:14:52.409306] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:18:19.437109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:18:19.437157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:18:19.437235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:14:53.814250 (xid=0x1257x) >[2012-10-01 12:18:19.437256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:18:19.437272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:21:22.868890] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:21:22.872333] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:21:22.875630] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:21:46.465105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:21:46.465151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:21:46.465215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:18:20.842621 (xid=0x1258x) >[2012-10-01 12:21:46.465236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:21:46.465252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:25:13.495105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:25:13.495159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:25:13.495226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:21:47.872147 (xid=0x1259x) >[2012-10-01 12:25:13.495248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:25:13.495264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:28:40.523159] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:28:40.523204] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:28:40.523271] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:25:14.899867 (xid=0x1260x) >[2012-10-01 12:28:40.523303] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:28:40.523320] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:31:22.951403] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:31:22.954377] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:31:22.956632] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:32:07.551130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:32:07.551187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:32:07.551262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:28:41.928480 (xid=0x1261x) >[2012-10-01 12:32:07.551284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:32:07.551301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:35:34.582129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:35:34.582178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:35:34.582251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:32:08.959251 (xid=0x1262x) >[2012-10-01 12:35:34.583065] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:35:34.583103] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:39:01.610127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:39:01.610179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:39:01.610248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:35:35.987076 (xid=0x1263x) >[2012-10-01 12:39:01.610269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:39:01.610298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:41:23.035255] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:41:23.038250] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:41:23.040690] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:42:28.637068] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:42:28.637115] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:42:28.637180] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:39:03.014513 (xid=0x1264x) >[2012-10-01 12:42:28.637201] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:42:28.637217] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:45:55.667101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:45:55.667152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:45:55.667220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:42:30.044790 (xid=0x1265x) >[2012-10-01 12:45:55.667241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:45:55.667258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:49:22.695120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:49:22.695168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:49:22.695234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:45:57.072411 (xid=0x1266x) >[2012-10-01 12:49:22.695256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:49:22.695272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:51:23.119957] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:51:23.122851] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:51:23.125014] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 12:52:49.725104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:52:49.725154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:52:49.725221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:49:24.102504 (xid=0x1267x) >[2012-10-01 12:52:49.725242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:52:49.725259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:56:16.755110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:56:16.755161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:56:16.755228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:52:51.132182 (xid=0x1268x) >[2012-10-01 12:56:16.755249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:56:16.755266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 12:59:43.782108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 12:59:43.782159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 12:59:43.782226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:56:18.159657 (xid=0x1269x) >[2012-10-01 12:59:43.782248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 12:59:43.782264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:01:23.203554] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:01:23.207106] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:01:23.209842] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:03:10.810157] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:03:10.810206] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:03:10.810273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 12:59:45.187196 (xid=0x1270x) >[2012-10-01 13:03:10.810293] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:03:10.810310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:06:37.841091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:06:37.841140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:06:37.841206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:03:12.217880 (xid=0x1271x) >[2012-10-01 13:06:37.841228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:06:37.841245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:10:04.875132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:10:04.875181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:10:04.875253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:06:39.252670 (xid=0x1272x) >[2012-10-01 13:10:04.875275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:10:04.875292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:11:23.294084] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:11:23.296642] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:11:23.298968] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:13:31.903108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:13:31.903168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:13:31.903238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:10:06.280183 (xid=0x1273x) >[2012-10-01 13:13:31.903259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:13:31.903275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:16:58.935105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:16:58.935154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:16:58.935222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:13:33.312613 (xid=0x1274x) >[2012-10-01 13:16:58.935243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:16:58.935260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:20:25.965119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:20:25.965168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:20:25.965235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:17:00.342066 (xid=0x1275x) >[2012-10-01 13:20:25.965257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:20:25.965273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:21:23.383268] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:21:23.386276] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:21:23.388704] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:23:52.995122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:23:52.995174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:23:52.995241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:20:27.372095 (xid=0x1276x) >[2012-10-01 13:23:52.995275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:23:52.995293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:27:20.027115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:27:20.027166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:27:20.027232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:23:54.404125 (xid=0x1277x) >[2012-10-01 13:27:20.027254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:27:20.027271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:30:47.057131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:30:47.057180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:30:47.057247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:27:21.433816 (xid=0x1278x) >[2012-10-01 13:30:47.057269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:30:47.057285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:31:23.472853] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:31:23.475958] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:31:23.479084] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:34:14.087114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:34:14.087164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:34:14.087235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:30:48.464045 (xid=0x1279x) >[2012-10-01 13:34:14.087267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:34:14.087285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:37:41.120156] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:37:41.120207] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:37:41.120275] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:34:15.497215 (xid=0x1280x) >[2012-10-01 13:37:41.120297] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:37:41.120314] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:41:08.149100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:41:08.149150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:41:08.149219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:37:42.526646 (xid=0x1281x) >[2012-10-01 13:41:08.149240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:41:08.149257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:41:23.563385] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:41:23.566380] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:41:23.568780] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:44:35.179034] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:44:35.179083] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:44:35.179149] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:41:09.556403 (xid=0x1282x) >[2012-10-01 13:44:35.179171] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:44:35.179187] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:48:02.212096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:48:02.212155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:48:02.212221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:44:36.589050 (xid=0x1283x) >[2012-10-01 13:48:02.212242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:48:02.212258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:51:23.647160] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:51:23.650268] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:51:23.652755] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 13:51:29.240108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:51:29.240150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:51:29.240225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:48:03.617729 (xid=0x1284x) >[2012-10-01 13:51:29.241103] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:51:29.241139] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:54:56.273102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:54:56.273155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:54:56.273230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:51:30.650082 (xid=0x1285x) >[2012-10-01 13:54:56.274008] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:54:56.274043] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 13:58:23.302140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 13:58:23.302192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 13:58:23.302268] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:54:57.679004 (xid=0x1286x) >[2012-10-01 13:58:23.303034] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 13:58:23.303068] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:01:23.733839] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:01:23.737336] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:01:23.740078] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:01:50.331146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:01:50.331191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:01:50.331256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 13:58:24.708191 (xid=0x1287x) >[2012-10-01 14:01:50.331277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:01:50.331293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:05:17.361105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:05:17.361162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:05:17.361236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:01:51.738682 (xid=0x1288x) >[2012-10-01 14:05:17.361258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:05:17.361275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:08:44.391053] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:08:44.391101] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:08:44.391168] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:05:18.768280 (xid=0x1289x) >[2012-10-01 14:08:44.391190] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:08:44.391221] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:11:23.822525] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:11:23.825624] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:11:23.827819] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:12:11.420089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:12:11.420136] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:12:11.420203] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:08:45.798043 (xid=0x1290x) >[2012-10-01 14:12:11.420224] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:12:11.420240] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:15:38.452126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:15:38.452176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:15:38.452242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:12:12.829607 (xid=0x1291x) >[2012-10-01 14:15:38.452263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:15:38.452280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:19:05.483109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:19:05.483158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:19:05.483225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:15:39.860371 (xid=0x1292x) >[2012-10-01 14:19:05.483247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:19:05.483264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:21:23.912388] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:21:23.915133] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:21:23.917487] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:22:32.514117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:22:32.514167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:22:32.514233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:19:06.890863 (xid=0x1293x) >[2012-10-01 14:22:32.514254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:22:32.514271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:25:59.545122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:25:59.545171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:25:59.545238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:22:33.922225 (xid=0x1294x) >[2012-10-01 14:25:59.545260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:25:59.545277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:29:26.576091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:29:26.576143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:29:26.576210] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:26:00.953009 (xid=0x1295x) >[2012-10-01 14:29:26.576232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:29:26.576249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:31:24.001131] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:31:24.004248] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:31:24.007418] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:32:53.606084] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:32:53.606135] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:32:53.606200] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:29:27.983126 (xid=0x1296x) >[2012-10-01 14:32:53.606221] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:32:53.606238] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:36:20.635123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:36:20.635172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:36:20.635239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:32:55.012550 (xid=0x1297x) >[2012-10-01 14:36:20.635260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:36:20.635277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:39:47.664135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:39:47.664184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:39:47.664252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:36:22.041215 (xid=0x1298x) >[2012-10-01 14:39:47.664273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:39:47.664290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:41:24.090144] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:41:24.093600] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:41:24.096043] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:43:14.695122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:43:14.695172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:43:14.695240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:39:49.072566 (xid=0x1299x) >[2012-10-01 14:43:14.695262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:43:14.695279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:46:41.730097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:46:41.730147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:46:41.730214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:43:16.107279 (xid=0x1300x) >[2012-10-01 14:46:41.730236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:46:41.730253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:50:11.761118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:50:11.761184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:50:11.761261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:46:46.138132 (xid=0x1301x) >[2012-10-01 14:50:11.761283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:50:11.761301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:51:24.181044] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:51:24.184100] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:51:24.186505] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 14:53:38.791133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:53:38.791187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:53:38.791268] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:50:13.167901 (xid=0x1302x) >[2012-10-01 14:53:38.791289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:53:38.791306] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 14:57:05.821131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 14:57:05.821181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 14:57:05.821247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:53:40.197921 (xid=0x1303x) >[2012-10-01 14:57:05.821269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 14:57:05.821285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:00:32.849119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:00:32.849169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:00:32.849237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 14:57:07.226484 (xid=0x1304x) >[2012-10-01 15:00:32.849259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:00:32.849276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:01:24.266076] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:01:24.268970] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:01:24.271365] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:03:59.878107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:03:59.878157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:03:59.878224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:00:34.255180 (xid=0x1305x) >[2012-10-01 15:03:59.878256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:03:59.878274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:07:26.908102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:07:26.908156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:07:26.908224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:04:01.285503 (xid=0x1306x) >[2012-10-01 15:07:26.908245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:07:26.908262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:10:53.937104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:10:53.937157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:10:53.937224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:07:28.314260 (xid=0x1307x) >[2012-10-01 15:10:53.937245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:10:53.937262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:11:24.350992] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:11:24.353680] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:11:24.356230] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:14:20.966099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:14:20.966150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:14:20.966219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:10:55.342912 (xid=0x1308x) >[2012-10-01 15:14:20.966240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:14:20.966257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:17:47.996108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:17:47.996171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:17:47.996238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:14:22.373532 (xid=0x1309x) >[2012-10-01 15:17:47.996259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:17:47.996275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:21:15.024098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:21:15.024149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:21:15.024219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:17:49.401407 (xid=0x1310x) >[2012-10-01 15:21:15.024242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:21:15.024259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:21:24.435200] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:21:24.437893] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:21:24.440305] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:24:42.052107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:24:42.052158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:24:42.052225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:21:16.429365 (xid=0x1311x) >[2012-10-01 15:24:42.052246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:24:42.052262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:28:09.083108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:28:09.083158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:28:09.083225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:24:43.459893 (xid=0x1312x) >[2012-10-01 15:28:09.083261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:28:09.083278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:31:24.518732] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:31:24.522198] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:31:24.525095] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:31:36.112134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:31:36.112182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:31:36.112246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:28:10.489347 (xid=0x1313x) >[2012-10-01 15:31:36.112266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:31:36.112283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:35:03.145126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:35:03.145172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:35:03.145238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:31:37.522260 (xid=0x1314x) >[2012-10-01 15:35:03.145259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:35:03.145275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:38:30.174120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:38:30.174173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:38:30.174240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:35:04.551038 (xid=0x1315x) >[2012-10-01 15:38:30.174271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:38:30.174289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:41:24.607899] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:41:24.611257] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:41:24.614461] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:41:57.204132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:41:57.204181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:41:57.204247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:38:31.581371 (xid=0x1316x) >[2012-10-01 15:41:57.204268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:41:57.204284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:45:24.237121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:45:24.237172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:45:24.237238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:41:58.614151 (xid=0x1317x) >[2012-10-01 15:45:24.237259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:45:24.237276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:48:51.268102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:48:51.268150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:48:51.268218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:45:25.645279 (xid=0x1318x) >[2012-10-01 15:48:51.268239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:48:51.268256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:51:24.696649] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:51:24.699082] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:51:24.701346] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 15:52:18.298131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:52:18.298191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:52:18.298265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:48:52.675027 (xid=0x1319x) >[2012-10-01 15:52:18.298288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:52:18.298304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:55:45.328122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:55:45.328171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:55:45.328237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:52:19.705230 (xid=0x1320x) >[2012-10-01 15:55:45.328259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:55:45.328275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 15:59:12.357097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 15:59:12.357157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 15:59:12.357232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:55:46.734265 (xid=0x1321x) >[2012-10-01 15:59:12.357255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 15:59:12.357272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:01:24.786012] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:01:24.788988] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:01:24.791392] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:02:39.387078] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:02:39.387128] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:02:39.387194] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 15:59:13.764267 (xid=0x1322x) >[2012-10-01 16:02:39.387216] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:02:39.387233] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:06:06.420119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:06:06.420173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:06:06.420237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:02:40.797406 (xid=0x1323x) >[2012-10-01 16:06:06.420258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:06:06.420275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:09:33.451076] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:09:33.451129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:09:33.451198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:06:07.828237 (xid=0x1324x) >[2012-10-01 16:09:33.451220] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:09:33.451236] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:11:24.876057] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:11:24.879609] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:11:24.882455] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:13:00.481117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:13:00.481171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:13:00.481239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:09:34.858070 (xid=0x1325x) >[2012-10-01 16:13:00.481260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:13:00.481276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:16:27.513104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:16:27.513154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:16:27.513222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:13:01.890397 (xid=0x1326x) >[2012-10-01 16:16:27.513244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:16:27.513261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:19:54.543106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:19:54.543157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:19:54.543225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:16:28.920037 (xid=0x1327x) >[2012-10-01 16:19:54.543246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:19:54.543262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:21:24.966689] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:21:24.969941] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:21:24.972403] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:23:21.572102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:23:21.572154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:23:21.572233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:19:55.949764 (xid=0x1328x) >[2012-10-01 16:23:21.572254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:23:21.572269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:26:48.605121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:26:48.605170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:26:48.605237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:23:22.982771 (xid=0x1329x) >[2012-10-01 16:26:48.605258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:26:48.605275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:30:15.635111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:30:15.635169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:30:15.635243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:26:50.012592 (xid=0x1330x) >[2012-10-01 16:30:15.635265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:30:15.635282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:31:25.055638] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:31:25.058584] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:31:25.061205] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:33:42.666103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:33:42.666151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:33:42.666215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:30:17.043051 (xid=0x1331x) >[2012-10-01 16:33:42.666247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:33:42.666265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:37:09.697106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:37:09.697157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:37:09.697223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:33:44.074048 (xid=0x1332x) >[2012-10-01 16:37:09.697245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:37:09.697262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:40:36.726108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:40:36.726156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:40:36.726223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:37:11.103757 (xid=0x1333x) >[2012-10-01 16:40:36.726246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:40:36.726263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:41:25.143661] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:41:25.146746] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:41:25.149365] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:44:03.756105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:44:03.756155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:44:03.756221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:40:38.133608 (xid=0x1334x) >[2012-10-01 16:44:03.756243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:44:03.756259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:47:30.787121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:47:30.787172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:47:30.787242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:44:05.163999 (xid=0x1335x) >[2012-10-01 16:47:30.787263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:47:30.787279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:50:57.815111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:50:57.815162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:50:57.815230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:47:32.192653 (xid=0x1336x) >[2012-10-01 16:50:57.815251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:50:57.815268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:51:25.228657] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:51:25.232078] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:51:25.235151] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 16:54:24.844053] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:54:24.844114] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:54:24.844183] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:50:59.220874 (xid=0x1337x) >[2012-10-01 16:54:24.844205] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:54:24.844221] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 16:57:51.873102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 16:57:51.873152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 16:57:51.873229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:54:26.250669 (xid=0x1338x) >[2012-10-01 16:57:51.873251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 16:57:51.873266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:01:18.903106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:01:18.903158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:01:18.903228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 16:57:53.279892 (xid=0x1339x) >[2012-10-01 17:01:18.903250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:01:18.903267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:01:25.314149] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:01:25.316909] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:01:25.319413] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:04:45.932114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:04:45.932164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:04:45.932231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:01:20.309018 (xid=0x1340x) >[2012-10-01 17:04:45.932252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:04:45.932270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:08:12.963117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:08:12.963173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:08:12.963246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:04:47.339831 (xid=0x1341x) >[2012-10-01 17:08:12.964177] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:08:12.964211] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:11:25.395251] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:11:25.398557] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:11:25.401286] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:11:39.991106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:11:39.991152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:11:39.991216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:08:14.368639 (xid=0x1342x) >[2012-10-01 17:11:39.991236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:11:39.991252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:15:07.022104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:15:07.022154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:15:07.022222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:11:41.399262 (xid=0x1343x) >[2012-10-01 17:15:07.022243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:15:07.022260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:18:34.050121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:18:34.050172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:18:34.050240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:15:08.427126 (xid=0x1344x) >[2012-10-01 17:18:34.050261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:18:34.050278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:21:25.479303] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:21:25.482601] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:21:25.485153] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:22:01.078101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:22:01.078147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:22:01.078209] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:18:35.455257 (xid=0x1345x) >[2012-10-01 17:22:01.078230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:22:01.078246] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:25:28.109113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:25:28.109166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:25:28.109242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:22:02.486138 (xid=0x1346x) >[2012-10-01 17:25:28.110193] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:25:28.110230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:28:55.137108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:28:55.137156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:28:55.137223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:25:29.513842 (xid=0x1347x) >[2012-10-01 17:28:55.137245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:28:55.137261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:31:25.563439] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:31:25.566157] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:31:25.568348] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:32:22.165074] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:32:22.165133] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:32:22.165207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:28:56.541971 (xid=0x1348x) >[2012-10-01 17:32:22.165228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:32:22.165245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:35:49.196120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:35:49.196169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:35:49.196236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:32:23.573033 (xid=0x1349x) >[2012-10-01 17:35:49.196257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:35:49.196274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:39:16.224104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:39:16.224155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:39:16.224221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:35:50.601631 (xid=0x1350x) >[2012-10-01 17:39:16.224243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:39:16.224259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:41:25.649916] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:41:25.653252] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:41:25.656044] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:42:43.253105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:42:43.253152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:42:43.253216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:39:17.630555 (xid=0x1351x) >[2012-10-01 17:42:43.253237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:42:43.253253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:46:10.284117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:46:10.284165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:46:10.284231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:42:44.661323 (xid=0x1352x) >[2012-10-01 17:46:10.284252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:46:10.284268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:49:37.311110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:49:37.311159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:49:37.311228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:46:11.688407 (xid=0x1353x) >[2012-10-01 17:49:37.311249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:49:37.311265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:51:25.733762] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:51:25.736716] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:51:25.739250] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 17:53:04.339104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:53:04.339148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:53:04.339228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:49:38.716072 (xid=0x1354x) >[2012-10-01 17:53:04.339249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:53:04.339268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:56:31.370108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:56:31.370159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:56:31.370227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:53:05.747257 (xid=0x1355x) >[2012-10-01 17:56:31.370249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:56:31.370266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 17:59:58.398104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 17:59:58.398154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 17:59:58.398220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:56:32.774912 (xid=0x1356x) >[2012-10-01 17:59:58.398243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 17:59:58.398259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:01:25.816891] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:01:25.819787] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:01:25.822986] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:03:25.425118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:03:25.425176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:03:25.425251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 17:59:59.802405 (xid=0x1357x) >[2012-10-01 18:03:25.425285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:03:25.425303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:06:52.455102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:06:52.455152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:06:52.455217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:03:26.832791 (xid=0x1358x) >[2012-10-01 18:06:52.455239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:06:52.455256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:10:19.483110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:10:19.483167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:10:19.483241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:06:53.860651 (xid=0x1359x) >[2012-10-01 18:10:19.483263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:10:19.483280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:11:25.900752] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:11:25.903997] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:11:25.906485] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:13:46.511118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:13:46.511167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:13:46.511234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:10:20.888561 (xid=0x1360x) >[2012-10-01 18:13:46.511255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:13:46.511285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:17:13.541100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:17:13.541152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:17:13.541219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:13:47.918498 (xid=0x1361x) >[2012-10-01 18:17:13.541240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:17:13.541257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:20:40.569094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:20:40.569143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:20:40.569211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:17:14.946372 (xid=0x1362x) >[2012-10-01 18:20:40.569232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:20:40.569250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:21:25.983787] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:21:25.987405] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:21:25.989872] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:24:07.597140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:24:07.597189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:24:07.597257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:20:41.974038 (xid=0x1363x) >[2012-10-01 18:24:07.597279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:24:07.597296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:27:34.627074] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:27:34.627124] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:27:34.627206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:24:09.004005 (xid=0x1364x) >[2012-10-01 18:27:34.627228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:27:34.627245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:31:01.654117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:31:01.654171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:31:01.654238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:27:36.031219 (xid=0x1365x) >[2012-10-01 18:31:01.654260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:31:01.654276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:31:26.065978] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:31:26.068186] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:31:26.070072] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:34:28.681113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:34:28.681170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:34:28.681240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:31:03.058411 (xid=0x1366x) >[2012-10-01 18:34:28.681261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:34:28.681277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:37:55.711055] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:37:55.711105] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:37:55.711186] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:34:30.088081 (xid=0x1367x) >[2012-10-01 18:37:55.711223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:37:55.711241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:41:22.738119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:41:22.738168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:41:22.738237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:37:57.115215 (xid=0x1368x) >[2012-10-01 18:41:22.738258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:41:22.738275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:41:26.148897] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:41:26.152351] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:41:26.155394] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:44:49.766113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:44:49.766165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:44:49.766232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:41:24.143311 (xid=0x1369x) >[2012-10-01 18:44:49.766253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:44:49.766270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:48:16.796994] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:48:16.797042] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:48:16.797107] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:44:51.174657 (xid=0x1370x) >[2012-10-01 18:48:16.797128] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:48:16.797144] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:51:26.229273] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:51:26.232391] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:51:26.235378] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 18:51:43.826120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:51:43.826166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:51:43.826232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:48:18.202835 (xid=0x1371x) >[2012-10-01 18:51:43.826252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:51:43.826268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:55:10.856108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:55:10.856160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:55:10.856227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:51:45.233340 (xid=0x1372x) >[2012-10-01 18:55:10.856249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:55:10.856266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 18:58:37.885072] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 18:58:37.885119] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 18:58:37.885184] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:55:12.262008 (xid=0x1373x) >[2012-10-01 18:58:37.885206] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 18:58:37.885223] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:01:26.317448] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:01:26.320542] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:01:26.323382] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:02:04.914118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:02:04.914164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:02:04.914228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 18:58:39.290871 (xid=0x1374x) >[2012-10-01 19:02:04.914249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:02:04.914265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:05:31.948097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:05:31.948146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:05:31.948214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:02:06.324894 (xid=0x1375x) >[2012-10-01 19:05:31.948235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:05:31.948252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:08:58.979099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:08:58.979148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:08:58.979215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:05:33.356060 (xid=0x1376x) >[2012-10-01 19:08:58.979236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:08:58.979253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:11:26.408620] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:11:26.411573] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:11:26.414071] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:12:26.009100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:12:26.009148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:12:26.009213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:09:00.386407 (xid=0x1377x) >[2012-10-01 19:12:26.009235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:12:26.009251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:15:53.038040] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:15:53.038088] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:15:53.038156] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:12:27.415772 (xid=0x1378x) >[2012-10-01 19:15:53.038178] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:15:53.038195] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:19:20.068104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:19:20.068169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:19:20.068255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:15:54.444882 (xid=0x1379x) >[2012-10-01 19:19:20.068278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:19:20.068296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:21:26.492285] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:21:26.495353] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:21:26.498374] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:22:47.096112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:22:47.096171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:22:47.096236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:19:21.473441 (xid=0x1380x) >[2012-10-01 19:22:47.096256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:22:47.096271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:26:14.125034] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:26:14.125090] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:26:14.125160] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:22:48.502570 (xid=0x1381x) >[2012-10-01 19:26:14.125182] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:26:14.125198] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:29:41.154104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:29:41.154152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:29:41.154221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:26:15.531221 (xid=0x1382x) >[2012-10-01 19:29:41.154243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:29:41.154259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:31:26.578509] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:31:26.581661] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:31:26.584514] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:33:08.184120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:33:08.184166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:33:08.184235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:29:42.560978 (xid=0x1383x) >[2012-10-01 19:33:08.184266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:33:08.184284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:36:35.216122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:36:35.216172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:36:35.216240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:33:09.593476 (xid=0x1384x) >[2012-10-01 19:36:35.216262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:36:35.216280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:40:02.244143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:40:02.244193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:40:02.244261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:36:36.621664 (xid=0x1385x) >[2012-10-01 19:40:02.244283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:40:02.244300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:41:26.664082] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:41:26.667080] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:41:26.670212] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:43:29.273115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:43:29.273170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:43:29.273244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:40:03.650029 (xid=0x1386x) >[2012-10-01 19:43:29.273266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:43:29.273296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:46:59.308121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:46:59.308172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:46:59.308240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:43:33.685138 (xid=0x1387x) >[2012-10-01 19:46:59.308262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:46:59.308278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:50:26.337167] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:50:26.337217] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:50:26.337286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:47:00.714605 (xid=0x1388x) >[2012-10-01 19:50:26.337308] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:50:26.337325] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:51:26.755266] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:51:26.758378] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:51:26.760524] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 19:53:53.366089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:53:53.366140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:53:53.366208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:50:27.743363 (xid=0x1389x) >[2012-10-01 19:53:53.366230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:53:53.366247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 19:57:20.397175] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 19:57:20.397226] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 19:57:20.397306] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:53:54.774023 (xid=0x1390x) >[2012-10-01 19:57:20.397328] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 19:57:20.397345] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:00:47.425108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:00:47.425162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:00:47.425235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 19:57:21.802412 (xid=0x1391x) >[2012-10-01 20:00:47.425257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:00:47.425273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:01:26.841696] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:01:26.845174] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:01:26.848117] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:04:14.455091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:04:14.455141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:04:14.455207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:00:48.832407 (xid=0x1392x) >[2012-10-01 20:04:14.455228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:04:14.455245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:07:41.488106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:07:41.488156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:07:41.488224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:04:15.864891 (xid=0x1393x) >[2012-10-01 20:07:41.488256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:07:41.488273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:11:08.518075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:11:08.518122] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:11:08.518189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:07:42.895022 (xid=0x1394x) >[2012-10-01 20:11:08.518211] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:11:08.518227] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:11:26.931200] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:11:26.933889] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:11:26.936010] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:14:35.548111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:14:35.548161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:14:35.548228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:11:09.925076 (xid=0x1395x) >[2012-10-01 20:14:35.548250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:14:35.548266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:18:02.580105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:18:02.580154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:18:02.580228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:14:36.957341 (xid=0x1396x) >[2012-10-01 20:18:02.580975] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:18:02.581042] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:21:27.020503] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:21:27.023870] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:21:27.026528] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:21:29.611123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:21:29.611161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:21:29.611228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:18:03.988171 (xid=0x1397x) >[2012-10-01 20:21:29.611249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:21:29.611266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:24:56.645116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:24:56.645167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:24:56.645232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:21:31.022550 (xid=0x1398x) >[2012-10-01 20:24:56.645253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:24:56.645269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:28:23.677106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:28:23.677157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:28:23.677224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:24:58.053816 (xid=0x1399x) >[2012-10-01 20:28:23.677247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:28:23.677263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:31:27.110340] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:31:27.113060] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:31:27.115350] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:31:50.705097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:31:50.705138] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:31:50.705202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:28:25.082381 (xid=0x1400x) >[2012-10-01 20:31:50.705222] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:31:50.705237] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:35:17.739111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:35:17.739162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:35:17.739230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:31:52.116574 (xid=0x1401x) >[2012-10-01 20:35:17.739252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:35:17.739268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:38:44.772102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:38:44.772147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:38:44.772213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:35:19.148983 (xid=0x1402x) >[2012-10-01 20:38:44.772234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:38:44.772250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:41:27.205931] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:41:27.209171] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:41:27.211817] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:42:11.802096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:42:11.802141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:42:11.802205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:38:46.179171 (xid=0x1403x) >[2012-10-01 20:42:11.802225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:42:11.802241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:45:38.837111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:45:38.837160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:45:38.837227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:42:13.214037 (xid=0x1404x) >[2012-10-01 20:45:38.837248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:45:38.837265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:49:05.870115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:49:05.870164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:49:05.870228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:45:40.247030 (xid=0x1405x) >[2012-10-01 20:49:05.870249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:49:05.870265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:51:27.297293] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:51:27.300882] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:51:27.303489] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 20:52:32.898112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:52:32.898184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:52:32.898259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:49:07.275684 (xid=0x1406x) >[2012-10-01 20:52:32.898280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:52:32.898296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:55:59.930120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:55:59.930170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:55:59.930238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:52:34.307711 (xid=0x1407x) >[2012-10-01 20:55:59.930259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:55:59.930276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 20:59:26.962115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 20:59:26.962169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 20:59:26.962238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:56:01.339696 (xid=0x1408x) >[2012-10-01 20:59:26.962259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 20:59:26.962276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:01:27.390397] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:01:27.393717] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:01:27.396826] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:02:53.994079] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:02:53.994127] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:02:53.994194] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 20:59:28.371051 (xid=0x1409x) >[2012-10-01 21:02:53.994228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:02:53.994246] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:06:21.025089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:06:21.025138] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:06:21.025206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:02:55.402600 (xid=0x1410x) >[2012-10-01 21:06:21.025227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:06:21.025244] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:09:48.054104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:09:48.054151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:09:48.054219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:06:22.431316 (xid=0x1411x) >[2012-10-01 21:09:48.054240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:09:48.054257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:11:27.480450] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:11:27.483927] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:11:27.486464] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:13:15.085094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:13:15.085140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:13:15.085207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:09:49.461911 (xid=0x1412x) >[2012-10-01 21:13:15.085239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:13:15.085256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:16:42.119100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:16:42.119151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:16:42.119215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:13:16.496669 (xid=0x1413x) >[2012-10-01 21:16:42.119237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:16:42.119254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:20:09.149109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:20:09.149158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:20:09.149226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:16:43.526311 (xid=0x1414x) >[2012-10-01 21:20:09.149247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:20:09.149264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:21:27.570622] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:21:27.573750] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:21:27.576311] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:23:36.180092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:23:36.180142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:23:36.180210] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:20:10.557105 (xid=0x1415x) >[2012-10-01 21:23:36.180231] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:23:36.180248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:27:03.211111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:27:03.211169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:27:03.211243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:23:37.588105 (xid=0x1416x) >[2012-10-01 21:27:03.211988] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:27:03.212024] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:30:30.241117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:30:30.241176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:30:30.241251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:27:04.618156 (xid=0x1417x) >[2012-10-01 21:30:30.241273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:30:30.241291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:31:27.661548] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:31:27.664575] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:31:27.667067] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:33:57.273114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:33:57.273169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:33:57.273235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:30:31.649902 (xid=0x1418x) >[2012-10-01 21:33:57.273256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:33:57.273273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:37:24.305098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:37:24.305145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:37:24.305211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:33:58.682127 (xid=0x1419x) >[2012-10-01 21:37:24.305243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:37:24.305260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:40:51.334121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:40:51.334171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:40:51.334238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:37:25.711680 (xid=0x1420x) >[2012-10-01 21:40:51.334260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:40:51.334277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:41:27.750299] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:41:27.753951] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:41:27.756326] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:44:18.365110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:44:18.365162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:44:18.365230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:40:52.741664 (xid=0x1421x) >[2012-10-01 21:44:18.365251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:44:18.365269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:47:45.396106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:47:45.396157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:47:45.396225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:44:19.773063 (xid=0x1422x) >[2012-10-01 21:47:45.396246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:47:45.396273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:51:12.426108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:51:12.426158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:51:12.426225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:47:46.803213 (xid=0x1423x) >[2012-10-01 21:51:12.426247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:51:12.426264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:51:27.839920] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:51:27.842976] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:51:27.845318] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 21:54:39.456099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:54:39.456150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:54:39.456218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:51:13.833720 (xid=0x1424x) >[2012-10-01 21:54:39.456239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:54:39.456256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 21:58:06.489100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 21:58:06.489153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 21:58:06.489220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:54:40.866286 (xid=0x1425x) >[2012-10-01 21:58:06.489242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 21:58:06.489258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:01:27.926448] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:01:27.929719] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:01:27.932466] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:01:33.519120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:01:33.519162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:01:33.519226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 21:58:07.896026 (xid=0x1426x) >[2012-10-01 22:01:33.519247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:01:33.519262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:05:00.552118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:05:00.552164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:05:00.552229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:01:34.929346 (xid=0x1427x) >[2012-10-01 22:05:00.552250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:05:00.552266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:08:27.582101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:08:27.582161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:08:27.582245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:05:01.959213 (xid=0x1428x) >[2012-10-01 22:08:27.583163] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:08:27.583207] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:11:28.017850] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:11:28.020795] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:11:28.023299] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:11:54.613111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:11:54.613158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:11:54.613222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:08:28.989867 (xid=0x1429x) >[2012-10-01 22:11:54.613242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:11:54.613258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:15:21.645126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:15:21.645177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:15:21.645242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:11:56.022829 (xid=0x1430x) >[2012-10-01 22:15:21.645264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:15:21.645281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:18:48.677115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:18:48.677168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:18:48.677236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:15:23.054636 (xid=0x1431x) >[2012-10-01 22:18:48.677257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:18:48.677274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:21:28.113562] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:21:28.116177] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:21:28.118477] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:22:15.710101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:22:15.710147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:22:15.710211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:18:50.086967 (xid=0x1432x) >[2012-10-01 22:22:15.710232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:22:15.710248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:25:42.745114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:25:42.745165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:25:42.745230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:22:17.122077 (xid=0x1433x) >[2012-10-01 22:25:42.745251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:25:42.745268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:29:09.777107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:29:09.777155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:29:09.777221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:25:44.154067 (xid=0x1434x) >[2012-10-01 22:29:09.777243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:29:09.777259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:31:28.208437] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:31:28.211927] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:31:28.215120] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:32:36.808103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:32:36.808150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:32:36.808229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:29:11.185124 (xid=0x1435x) >[2012-10-01 22:32:36.808250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:32:36.808265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:36:03.841115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:36:03.841161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:36:03.841225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:32:38.218681 (xid=0x1436x) >[2012-10-01 22:36:03.841246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:36:03.841262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:39:30.872095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:39:30.872152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:39:30.872223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:36:05.249715 (xid=0x1437x) >[2012-10-01 22:39:30.872245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:39:30.872262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:41:28.300402] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:41:28.303957] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:41:28.306438] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:42:57.903117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:42:57.903165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:42:57.903232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:39:32.280340 (xid=0x1438x) >[2012-10-01 22:42:57.903265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:42:57.903282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:46:24.936100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:46:24.936149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:46:24.936218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:42:59.313783 (xid=0x1439x) >[2012-10-01 22:46:24.936240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:46:24.936257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:49:51.967100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:49:51.967149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:49:51.967216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:46:26.344306 (xid=0x1440x) >[2012-10-01 22:49:51.967238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:49:51.967255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:51:28.388845] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:51:28.392162] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:51:28.395166] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 22:53:18.995110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:53:18.995165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:53:18.995241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:49:53.372280 (xid=0x1441x) >[2012-10-01 22:53:18.996011] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:53:18.996049] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 22:56:46.026112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 22:56:46.026174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 22:56:46.026248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:53:20.403552 (xid=0x1442x) >[2012-10-01 22:56:46.027015] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 22:56:46.027053] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:00:13.054117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:00:13.054169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:00:13.054238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 22:56:47.431409 (xid=0x1443x) >[2012-10-01 23:00:13.054260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:00:13.054277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:01:28.472947] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:01:28.476430] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:01:28.478911] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:03:40.082131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:03:40.082182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:03:40.082247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:00:14.459800 (xid=0x1444x) >[2012-10-01 23:03:40.082268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:03:40.082284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:07:07.113113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:07:07.113167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:07:07.113242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:03:41.490308 (xid=0x1445x) >[2012-10-01 23:07:07.114014] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:07:07.114047] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:10:34.141113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:10:34.141163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:10:34.141231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:07:08.518081 (xid=0x1446x) >[2012-10-01 23:10:34.141252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:10:34.141269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:11:28.556346] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:11:28.559306] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:11:28.561647] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:14:01.169068] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:14:01.169116] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:14:01.169180] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:10:35.546239 (xid=0x1447x) >[2012-10-01 23:14:01.169201] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:14:01.169217] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:17:28.200100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:17:28.200146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:17:28.200213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:14:02.577075 (xid=0x1448x) >[2012-10-01 23:17:28.200245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:17:28.200263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:20:55.231114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:20:55.231168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:20:55.231234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:17:29.608479 (xid=0x1449x) >[2012-10-01 23:20:55.231256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:20:55.231273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:21:28.646438] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:21:28.650208] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:21:28.652840] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:24:22.260104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:24:22.260156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:24:22.260220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:20:56.637408 (xid=0x1450x) >[2012-10-01 23:24:22.260241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:24:22.260257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:27:49.292085] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:27:49.292133] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:27:49.292198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:24:23.669366 (xid=0x1451x) >[2012-10-01 23:27:49.292219] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:27:49.292236] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:31:16.321123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:31:16.321180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:31:16.321247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:27:50.698361 (xid=0x1452x) >[2012-10-01 23:31:16.321268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:31:16.321284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:31:28.733749] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:31:28.737028] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:31:28.739683] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:34:43.350108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:34:43.350160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:34:43.350228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:31:17.727774 (xid=0x1453x) >[2012-10-01 23:34:43.350249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:34:43.350266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:38:10.382122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:38:10.382172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:38:10.382241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:34:44.759378 (xid=0x1454x) >[2012-10-01 23:38:10.382262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:38:10.382279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:41:28.818698] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:41:28.821756] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:41:28.824067] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:41:37.412106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:41:37.412150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:41:37.412216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:38:11.789438 (xid=0x1455x) >[2012-10-01 23:41:37.412236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:41:37.412251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:45:04.444106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:45:04.444153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:45:04.444220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:41:38.821776 (xid=0x1456x) >[2012-10-01 23:45:04.444241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:45:04.444257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:48:31.474112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:48:31.474164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:48:31.474233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:45:05.851653 (xid=0x1457x) >[2012-10-01 23:48:31.474254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:48:31.474270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:51:28.909118] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:51:28.912334] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:51:28.915213] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-01 23:51:58.505118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:51:58.505159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:51:58.505224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:48:32.882219 (xid=0x1458x) >[2012-10-01 23:51:58.505243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:51:58.505259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:55:25.538107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:55:25.538163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:55:25.538235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:51:59.915416 (xid=0x1459x) >[2012-10-01 23:55:25.538257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:55:25.538273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-01 23:58:52.568109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-01 23:58:52.568161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-01 23:58:52.568228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:55:26.945491 (xid=0x1460x) >[2012-10-01 23:58:52.568249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-01 23:58:52.568266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:01:29.002475] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:01:29.005526] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:01:29.007701] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:02:19.599123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:02:19.599169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:02:19.599246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-01 23:58:53.976579 (xid=0x1461x) >[2012-10-02 00:02:19.599267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:02:19.599282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:05:46.633121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:05:46.633175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:05:46.633241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:02:21.010662 (xid=0x1462x) >[2012-10-02 00:05:46.633263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:05:46.633280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:09:13.665116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:09:13.665164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:09:13.665231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:05:48.042521 (xid=0x1463x) >[2012-10-02 00:09:13.665253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:09:13.665269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:11:29.097804] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:11:29.101423] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:11:29.103938] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:12:40.697101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:12:40.697159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:12:40.697236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:09:15.074354 (xid=0x1464x) >[2012-10-02 00:12:40.697272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:12:40.697293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:16:07.732109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:16:07.732159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:16:07.732226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:12:42.109413 (xid=0x1465x) >[2012-10-02 00:16:07.732248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:16:07.732265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:19:34.765059] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:19:34.765112] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:19:34.765181] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:16:09.141922 (xid=0x1466x) >[2012-10-02 00:19:34.765203] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:19:34.765219] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:21:29.195780] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:21:29.198462] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:21:29.200569] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:23:01.798125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:23:01.798172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:23:01.798239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:19:36.175318 (xid=0x1467x) >[2012-10-02 00:23:01.798260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:23:01.798277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:26:28.833118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:26:28.833168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:26:28.833234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:23:03.210528 (xid=0x1468x) >[2012-10-02 00:26:28.833256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:26:28.833274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:29:55.862112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:29:55.862161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:29:55.862236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:26:30.239285 (xid=0x1469x) >[2012-10-02 00:29:55.863290] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:29:55.863334] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:31:29.290232] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:31:29.292959] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:31:29.295331] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:33:22.894100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:33:22.894147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:33:22.894214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:29:57.271799 (xid=0x1470x) >[2012-10-02 00:33:22.894234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:33:22.894250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:36:49.930097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:36:49.930147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:36:49.930225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:33:24.307776 (xid=0x1471x) >[2012-10-02 00:36:49.930247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:36:49.930263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:40:16.960121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:40:16.960171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:40:16.960238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:36:51.337475 (xid=0x1472x) >[2012-10-02 00:40:16.960259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:40:16.960276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:41:29.385001] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:41:29.388258] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:41:29.390572] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:43:43.993132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:43:43.993180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:43:43.993245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:40:18.370682 (xid=0x1473x) >[2012-10-02 00:43:43.993266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:43:43.993282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:47:11.026111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:47:11.026160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:47:11.026226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:43:45.403742 (xid=0x1474x) >[2012-10-02 00:47:11.026259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:47:11.026276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:50:38.058127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:50:38.058181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:50:38.058251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:47:12.435094 (xid=0x1475x) >[2012-10-02 00:50:38.058273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:50:38.058290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:51:29.477048] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:51:29.479803] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:51:29.482156] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 00:54:05.088112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:54:05.088161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:54:05.088230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:50:39.465596 (xid=0x1476x) >[2012-10-02 00:54:05.088251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:54:05.088268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 00:57:32.122120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 00:57:32.122183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 00:57:32.122262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:54:06.499012 (xid=0x1477x) >[2012-10-02 00:57:32.122285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 00:57:32.122303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:00:59.152113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:00:59.152163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:00:59.152231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 00:57:33.529017 (xid=0x1478x) >[2012-10-02 01:00:59.152253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:00:59.152269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:01:29.565855] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:01:29.569146] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:01:29.571667] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:04:26.181065] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:04:26.181115] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:04:26.181179] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:01:00.558050 (xid=0x1479x) >[2012-10-02 01:04:26.181200] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:04:26.181216] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:07:53.212097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:07:53.212147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:07:53.212215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:04:27.589391 (xid=0x1480x) >[2012-10-02 01:07:53.212237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:07:53.212254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:11:20.242107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:11:20.242156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:11:20.242231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:07:54.619375 (xid=0x1481x) >[2012-10-02 01:11:20.243269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:11:20.243310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:11:29.653914] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:11:29.656585] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:11:29.658608] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:14:47.271121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:14:47.271172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:14:47.271237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:11:21.648749 (xid=0x1482x) >[2012-10-02 01:14:47.271258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:14:47.271274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:18:14.302110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:18:14.302166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:18:14.302241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:14:48.679425 (xid=0x1483x) >[2012-10-02 01:18:14.303007] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:18:14.303044] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:21:29.736420] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:21:29.739844] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:21:29.742444] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:21:41.331127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:21:41.331170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:21:41.331236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:18:15.707879 (xid=0x1484x) >[2012-10-02 01:21:41.331256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:21:41.331271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:25:08.363109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:25:08.363158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:25:08.363224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:21:42.740020 (xid=0x1485x) >[2012-10-02 01:25:08.363246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:25:08.363262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:28:35.391100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:28:35.391148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:28:35.391217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:25:09.768224 (xid=0x1486x) >[2012-10-02 01:28:35.391238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:28:35.391255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:31:29.827333] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:31:29.830025] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:31:29.832382] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:32:02.425099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:32:02.425146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:32:02.425222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:28:36.802660 (xid=0x1487x) >[2012-10-02 01:32:02.425243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:32:02.425258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:35:29.455098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:35:29.455147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:35:29.455213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:32:03.832217 (xid=0x1488x) >[2012-10-02 01:35:29.455235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:35:29.455252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:38:56.484094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:38:56.484147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:38:56.484214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:35:30.861074 (xid=0x1489x) >[2012-10-02 01:38:56.484236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:38:56.484252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:41:29.912058] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:41:29.915605] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:41:29.918839] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:42:23.512100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:42:23.512146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:42:23.512213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:38:57.889632 (xid=0x1490x) >[2012-10-02 01:42:23.512245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:42:23.512262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:45:50.543110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:45:50.543159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:45:50.543226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:42:24.920131 (xid=0x1491x) >[2012-10-02 01:45:50.543247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:45:50.543265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:49:17.573100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:49:17.573149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:49:17.573215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:45:51.950054 (xid=0x1492x) >[2012-10-02 01:49:17.573237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:49:17.573254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:51:30.003388] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:51:30.006666] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:51:30.009530] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 01:52:44.604110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:52:44.604164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:52:44.604237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:49:18.981504 (xid=0x1493x) >[2012-10-02 01:52:44.604259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:52:44.604290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:56:11.637104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:56:11.637157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:56:11.637224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:52:46.014287 (xid=0x1494x) >[2012-10-02 01:56:11.637246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:56:11.637263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 01:59:38.669104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 01:59:38.669161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 01:59:38.669236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:56:13.046413 (xid=0x1495x) >[2012-10-02 01:59:38.669258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 01:59:38.669275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:01:30.098824] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:01:30.101526] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:01:30.104006] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:03:05.702047] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:03:05.702102] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:03:05.702172] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 01:59:40.079052 (xid=0x1496x) >[2012-10-02 02:03:05.702193] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:03:05.702210] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:06:32.736121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:06:32.736182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:06:32.736270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:03:07.113019 (xid=0x1497x) >[2012-10-02 02:06:32.736291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:06:32.736308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:09:59.768096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:09:59.768145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:09:59.768211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:06:34.145128 (xid=0x1498x) >[2012-10-02 02:09:59.768233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:09:59.768249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:11:30.192677] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:11:30.195290] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:11:30.197641] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:13:26.798108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:13:26.798161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:13:26.798229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:10:01.175721 (xid=0x1499x) >[2012-10-02 02:13:26.798251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:13:26.798267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:16:53.832111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:16:53.832161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:16:53.832228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:13:28.209686 (xid=0x1500x) >[2012-10-02 02:16:53.832260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:16:53.832278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:20:20.864104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:20:20.864152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:20:20.864219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:16:55.241036 (xid=0x1501x) >[2012-10-02 02:20:20.864241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:20:20.864256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:21:30.286011] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:21:30.288889] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:21:30.291345] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:23:47.895108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:23:47.895171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:23:47.895256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:20:22.272387 (xid=0x1502x) >[2012-10-02 02:23:47.896199] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:23:47.896239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:27:14.927102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:27:14.927150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:27:14.927217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:23:49.304749 (xid=0x1503x) >[2012-10-02 02:27:14.927238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:27:14.927255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:30:41.957099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:30:41.957150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:30:41.957216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:27:16.334398 (xid=0x1504x) >[2012-10-02 02:30:41.957238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:30:41.957255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:31:30.373837] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:31:30.376953] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:31:30.379447] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:34:08.986101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:34:08.986155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:34:08.986231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:30:43.363295 (xid=0x1505x) >[2012-10-02 02:34:08.986261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:34:08.986279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:37:36.016118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:37:36.016168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:37:36.016235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:34:10.393448 (xid=0x1506x) >[2012-10-02 02:37:36.016256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:37:36.016273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:41:03.044125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:41:03.044173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:41:03.044252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:37:37.421427 (xid=0x1507x) >[2012-10-02 02:41:03.044273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:41:03.044290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:41:30.457226] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:41:30.459910] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:41:30.462318] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:44:30.073131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:44:30.073180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:44:30.073248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:41:04.450133 (xid=0x1508x) >[2012-10-02 02:44:30.073269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:44:30.073286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:47:57.103102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:47:57.103152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:47:57.103227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:44:31.480057 (xid=0x1509x) >[2012-10-02 02:47:57.104014] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:47:57.104051] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:51:24.131083] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:51:24.131138] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:51:24.131206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:47:58.508656 (xid=0x1510x) >[2012-10-02 02:51:24.131238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:51:24.131256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:51:30.541220] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:51:30.543770] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:51:30.546351] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 02:54:51.159137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:54:51.159188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:54:51.159262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:51:25.536221 (xid=0x1511x) >[2012-10-02 02:54:51.160073] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:54:51.160109] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 02:58:18.187141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 02:58:18.187192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 02:58:18.187258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:54:52.564742 (xid=0x1512x) >[2012-10-02 02:58:18.187280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 02:58:18.187297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:01:30.620609] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:01:30.623867] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:01:30.626757] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:01:45.215118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:01:45.215184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:01:45.215260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 02:58:19.592506 (xid=0x1513x) >[2012-10-02 03:01:45.215281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:01:45.215298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:05:12.246099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:05:12.246148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:05:12.246213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:01:46.623067 (xid=0x1514x) >[2012-10-02 03:05:12.246235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:05:12.246252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:08:39.274114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:08:39.274159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:08:39.274225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:05:13.651316 (xid=0x1515x) >[2012-10-02 03:08:39.274247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:08:39.274263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:11:30.703010] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:11:30.706000] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:11:30.708503] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:12:06.302101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:12:06.302151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:12:06.302219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:08:40.679112 (xid=0x1516x) >[2012-10-02 03:12:06.302259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:12:06.302276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:15:33.332099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:15:33.332149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:15:33.332217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:12:07.709570 (xid=0x1517x) >[2012-10-02 03:15:33.332239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:15:33.332256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:19:00.361058] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:19:00.361106] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:19:00.361172] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:15:34.738530 (xid=0x1518x) >[2012-10-02 03:19:00.361194] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:19:00.361210] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:21:30.790025] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:21:30.793298] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:21:30.796112] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:22:27.390098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:22:27.390142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:22:27.390206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:19:01.767703 (xid=0x1519x) >[2012-10-02 03:22:27.390227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:22:27.390255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:25:54.422132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:25:54.422184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:25:54.422250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:22:28.799555 (xid=0x1520x) >[2012-10-02 03:25:54.422271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:25:54.422288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:29:21.452101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:29:21.452149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:29:21.452213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:25:55.829101 (xid=0x1521x) >[2012-10-02 03:29:21.452234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:29:21.452256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:31:30.878038] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:31:30.881239] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:31:30.883937] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:32:48.481109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:32:48.481162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:32:48.481237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:29:22.858005 (xid=0x1522x) >[2012-10-02 03:32:48.481259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:32:48.481276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:36:15.513119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:36:15.513166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:36:15.513245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:32:49.890265 (xid=0x1523x) >[2012-10-02 03:36:15.513267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:36:15.513283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:39:42.544104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:39:42.544162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:39:42.544238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:36:16.920893 (xid=0x1524x) >[2012-10-02 03:39:42.544262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:39:42.544278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:41:30.970581] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:41:30.974059] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:41:30.977030] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:43:09.575103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:43:09.575151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:43:09.575216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:39:43.952287 (xid=0x1525x) >[2012-10-02 03:43:09.575236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:43:09.575252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:46:36.608121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:46:36.608172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:46:36.608241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:43:10.985675 (xid=0x1526x) >[2012-10-02 03:46:36.608273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:46:36.608292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:50:03.640106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:50:03.640156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:50:03.640225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:46:38.017367 (xid=0x1527x) >[2012-10-02 03:50:03.640247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:50:03.640264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:51:31.065067] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:51:31.068640] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:51:31.071713] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 03:53:30.672117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:53:30.672167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:53:30.672232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:50:05.049052 (xid=0x1528x) >[2012-10-02 03:53:30.672253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:53:30.672270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 03:56:57.704059] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 03:56:57.704110] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 03:56:57.704179] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:53:32.081690 (xid=0x1529x) >[2012-10-02 03:56:57.704201] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 03:56:57.704230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:00:24.735116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:00:24.735167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:00:24.735232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 03:56:59.112662 (xid=0x1530x) >[2012-10-02 04:00:24.735253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:00:24.735270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:01:31.157778] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:01:31.160776] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:01:31.163309] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:03:51.767108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:03:51.767156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:03:51.767222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:00:26.144318 (xid=0x1531x) >[2012-10-02 04:03:51.767243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:03:51.767259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:07:18.802090] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:07:18.802141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:07:18.802208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:03:53.179374 (xid=0x1532x) >[2012-10-02 04:07:18.802229] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:07:18.802245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:10:45.834102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:10:45.834154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:10:45.834233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:07:20.211293 (xid=0x1533x) >[2012-10-02 04:10:45.834255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:10:45.834271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:11:31.252663] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:11:31.255848] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:11:31.258853] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:14:12.866124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:14:12.866175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:14:12.866241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:10:47.242852 (xid=0x1534x) >[2012-10-02 04:14:12.866262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:14:12.866278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:17:39.898096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:17:39.898146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:17:39.898215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:14:14.275295 (xid=0x1535x) >[2012-10-02 04:17:39.898236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:17:39.898253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:21:06.929107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:21:06.929161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:21:06.929228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:17:41.306258 (xid=0x1536x) >[2012-10-02 04:21:06.929259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:21:06.929276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:21:31.344468] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:21:31.347954] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:21:31.350259] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:24:33.960107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:24:33.960158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:24:33.960226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:21:08.337357 (xid=0x1537x) >[2012-10-02 04:24:33.960247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:24:33.960264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:28:00.990081] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:28:00.990130] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:28:00.990196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:24:35.367798 (xid=0x1538x) >[2012-10-02 04:28:00.990218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:28:00.990234] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:31:28.019117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:31:28.019167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:31:28.019240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:28:02.396515 (xid=0x1539x) >[2012-10-02 04:31:28.019262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:31:28.019286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:31:31.429667] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:31:31.432321] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:31:31.434358] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:34:55.048116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:34:55.048163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:34:55.048230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:31:29.425417 (xid=0x1540x) >[2012-10-02 04:34:55.048252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:34:55.048270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:38:22.081108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:38:22.081160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:38:22.081226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:34:56.458174 (xid=0x1541x) >[2012-10-02 04:38:22.081247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:38:22.081264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:41:31.517530] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:41:31.520361] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:41:31.522643] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:41:49.111103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:41:49.111149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:41:49.111212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:38:23.488365 (xid=0x1542x) >[2012-10-02 04:41:49.111251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:41:49.111268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:45:16.143129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:45:16.143182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:45:16.143247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:41:50.520519 (xid=0x1543x) >[2012-10-02 04:45:16.143268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:45:16.143284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:48:43.173133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:48:43.173180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:48:43.173246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:45:17.550044 (xid=0x1544x) >[2012-10-02 04:48:43.173267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:48:43.173284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:51:31.605779] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:51:31.609066] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:51:31.611495] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 04:52:10.203124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:52:10.203173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:52:10.203240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:48:44.579891 (xid=0x1545x) >[2012-10-02 04:52:10.203272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:52:10.203289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:55:37.235125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:55:37.235178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:55:37.235245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:52:11.612696 (xid=0x1546x) >[2012-10-02 04:55:37.235266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:55:37.235284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 04:59:04.266109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 04:59:04.266156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 04:59:04.266220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:55:38.643338 (xid=0x1547x) >[2012-10-02 04:59:04.266247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 04:59:04.266263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:01:31.695037] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:01:31.697967] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:01:31.700295] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:02:31.296109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:02:31.296161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:02:31.296235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 04:59:05.673384 (xid=0x1548x) >[2012-10-02 05:02:31.297253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:02:31.297289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:05:58.326124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:05:58.326181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:05:58.326245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:02:32.703118 (xid=0x1549x) >[2012-10-02 05:05:58.326266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:05:58.326282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:09:25.355116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:09:25.355168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:09:25.355236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:05:59.732020 (xid=0x1550x) >[2012-10-02 05:09:25.355257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:09:25.355273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:11:31.781189] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:11:31.783596] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:11:31.785697] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:12:52.384132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:12:52.384182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:12:52.384249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:09:26.761277 (xid=0x1551x) >[2012-10-02 05:12:52.384271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:12:52.384287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:16:19.415111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:16:19.415159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:16:19.415225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:12:53.792184 (xid=0x1552x) >[2012-10-02 05:16:19.415259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:16:19.415277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:19:46.445129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:19:46.445179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:19:46.445244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:16:20.822036 (xid=0x1553x) >[2012-10-02 05:19:46.445265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:19:46.445282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:21:31.869507] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:21:31.871991] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:21:31.874055] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:23:13.475107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:23:13.475158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:23:13.475224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:19:47.852198 (xid=0x1554x) >[2012-10-02 05:23:13.475245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:23:13.475262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:26:40.511093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:26:40.511145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:26:40.511212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:23:14.888470 (xid=0x1555x) >[2012-10-02 05:26:40.511234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:26:40.511260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:30:07.541130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:30:07.541180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:30:07.541247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:26:41.918533 (xid=0x1556x) >[2012-10-02 05:30:07.541269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:30:07.541285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:31:31.965788] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:31:31.969109] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:31:31.972022] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:33:34.572094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:33:34.572146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:33:34.572214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:30:08.949182 (xid=0x1557x) >[2012-10-02 05:33:34.572236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:33:34.572253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:37:01.606115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:37:01.606165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:37:01.606231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:33:35.983105 (xid=0x1558x) >[2012-10-02 05:37:01.606254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:37:01.606270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:40:28.637130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:40:28.637179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:40:28.637257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:37:03.013861 (xid=0x1559x) >[2012-10-02 05:40:28.637279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:40:28.637295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:41:32.057059] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:41:32.060390] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:41:32.063103] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:43:55.667106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:43:55.667155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:43:55.667222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:40:30.044436 (xid=0x1560x) >[2012-10-02 05:43:55.667244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:43:55.667260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:47:22.700125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:47:22.700175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:47:22.700240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:43:57.077589 (xid=0x1561x) >[2012-10-02 05:47:22.700261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:47:22.700277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:50:49.731107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:50:49.731159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:50:49.731227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:47:24.108704 (xid=0x1562x) >[2012-10-02 05:50:49.731260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:50:49.731278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:51:32.147756] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:51:32.150421] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:51:32.152541] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 05:54:16.761113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:54:16.761163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:54:16.761228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:50:51.138505 (xid=0x1563x) >[2012-10-02 05:54:16.761249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:54:16.761265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 05:57:43.793115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 05:57:43.793164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 05:57:43.793231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:54:18.170676 (xid=0x1564x) >[2012-10-02 05:57:43.793253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 05:57:43.793270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:01:10.823125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:01:10.823172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:01:10.823242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 05:57:45.200763 (xid=0x1565x) >[2012-10-02 06:01:10.823263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:01:10.823293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:01:32.236342] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:01:32.239057] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:01:32.241310] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:04:37.853095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:04:37.853147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:04:37.853215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:01:12.229967 (xid=0x1566x) >[2012-10-02 06:04:37.853237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:04:37.853254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:08:04.884141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:08:04.884193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:08:04.884258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:04:39.261288 (xid=0x1567x) >[2012-10-02 06:08:04.884279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:08:04.884296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:11:31.915113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:11:31.915163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:11:31.915227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:08:06.292528 (xid=0x1568x) >[2012-10-02 06:11:31.915248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:11:31.915264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:11:32.324552] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:11:32.327518] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:11:32.330263] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:14:58.950133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:14:58.950180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:14:58.950247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:11:33.326846 (xid=0x1569x) >[2012-10-02 06:14:58.950269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:14:58.950286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:18:25.979118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:18:25.979167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:18:25.979233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:15:00.356184 (xid=0x1570x) >[2012-10-02 06:18:25.979254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:18:25.979271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:21:32.412227] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:21:32.415460] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:21:32.418023] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:21:53.007117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:21:53.007171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:21:53.007246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:18:27.384644 (xid=0x1571x) >[2012-10-02 06:21:53.007282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:21:53.007299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:25:20.039129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:25:20.039179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:25:20.039246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:21:54.416217 (xid=0x1572x) >[2012-10-02 06:25:20.039268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:25:20.039284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:28:47.068136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:28:47.068196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:28:47.068273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:25:21.445331 (xid=0x1573x) >[2012-10-02 06:28:47.068295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:28:47.068312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:31:32.499400] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:31:32.502499] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:31:32.504582] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:32:14.098080] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:32:14.098125] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:32:14.098187] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:28:48.474839 (xid=0x1574x) >[2012-10-02 06:32:14.098208] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:32:14.098224] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:35:41.127125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:35:41.127188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:35:41.127254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:32:15.504550 (xid=0x1575x) >[2012-10-02 06:35:41.127275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:35:41.127291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:39:08.156122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:39:08.156172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:39:08.156240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:35:42.533069 (xid=0x1576x) >[2012-10-02 06:39:08.156262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:39:08.156279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:41:32.583987] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:41:32.587192] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:41:32.589542] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:42:35.185114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:42:35.185157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:42:35.185221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:39:09.562304 (xid=0x1577x) >[2012-10-02 06:42:35.185242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:42:35.185259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:46:02.217136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:46:02.217187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:46:02.217252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:42:36.593997 (xid=0x1578x) >[2012-10-02 06:46:02.217283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:46:02.217301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:49:29.247121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:49:29.247170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:49:29.247235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:46:03.623839 (xid=0x1579x) >[2012-10-02 06:49:29.247257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:49:29.247273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:51:32.671974] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:51:32.675494] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:51:32.678189] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 06:52:56.276113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:52:56.276164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:52:56.276232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:49:30.652996 (xid=0x1580x) >[2012-10-02 06:52:56.276253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:52:56.276269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:56:23.305137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:56:23.305187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:56:23.305254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:52:57.682393 (xid=0x1581x) >[2012-10-02 06:56:23.305286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:56:23.305304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 06:59:50.333136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 06:59:50.333187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 06:59:50.333254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:56:24.710235 (xid=0x1582x) >[2012-10-02 06:59:50.333275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 06:59:50.333292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:01:32.754649] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:01:32.757769] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:01:32.760075] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:03:17.361134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:03:17.361191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:03:17.361259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 06:59:51.738362 (xid=0x1583x) >[2012-10-02 07:03:17.361280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:03:17.361297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:06:44.391138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:06:44.391187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:06:44.391253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:03:18.768037 (xid=0x1584x) >[2012-10-02 07:06:44.391275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:06:44.391291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:10:11.419074] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:10:11.419134] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:10:11.419202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:06:45.796544 (xid=0x1585x) >[2012-10-02 07:10:11.419223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:10:11.419240] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:11:32.840374] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:11:32.843254] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:11:32.845966] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:13:38.447127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:13:38.447170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:13:38.447233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:10:12.824803 (xid=0x1586x) >[2012-10-02 07:13:38.447253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:13:38.447269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:17:05.486120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:17:05.486168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:17:05.486234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:13:39.862851 (xid=0x1587x) >[2012-10-02 07:17:05.486255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:17:05.486271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:20:32.514138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:20:32.514187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:20:32.514251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:17:06.891125 (xid=0x1588x) >[2012-10-02 07:20:32.514284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:20:32.514302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:21:32.934051] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:21:32.937237] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:21:32.939693] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:23:59.545104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:23:59.545151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:23:59.545216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:20:33.922064 (xid=0x1589x) >[2012-10-02 07:23:59.545236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:23:59.545251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:27:26.577113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:27:26.577163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:27:26.577230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:24:00.954094 (xid=0x1590x) >[2012-10-02 07:27:26.577251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:27:26.577268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:30:53.608124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:30:53.608178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:30:53.608252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:27:27.985384 (xid=0x1591x) >[2012-10-02 07:30:53.608273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:30:53.608303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:31:33.025979] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:31:33.028657] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:31:33.030916] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:34:20.639119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:34:20.639168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:34:20.639235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:30:55.016678 (xid=0x1592x) >[2012-10-02 07:34:20.639257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:34:20.639274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:37:47.672116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:37:47.672172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:37:47.672245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:34:22.048865 (xid=0x1593x) >[2012-10-02 07:37:47.672268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:37:47.672285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:41:14.706107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:41:14.706157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:41:14.706223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:37:49.083008 (xid=0x1594x) >[2012-10-02 07:41:14.706244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:41:14.706261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:41:33.127464] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:41:33.130997] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:41:33.133955] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:44:41.743102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:44:41.743150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:44:41.743218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:41:16.120087 (xid=0x1595x) >[2012-10-02 07:44:41.743239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:44:41.743256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:48:08.778142] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:48:08.778188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:48:08.778255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:44:43.155226 (xid=0x1596x) >[2012-10-02 07:48:08.778277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:48:08.778293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:51:33.222280] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:51:33.225154] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:51:33.227572] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 07:51:35.811130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:51:35.811168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:51:35.811229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:48:10.188756 (xid=0x1597x) >[2012-10-02 07:51:35.811261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:51:35.811278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:55:02.847114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:55:02.847162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:55:02.847229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:51:37.224147 (xid=0x1598x) >[2012-10-02 07:55:02.847250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:55:02.847267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 07:58:29.881066] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 07:58:29.881113] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 07:58:29.881178] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:55:04.258022 (xid=0x1599x) >[2012-10-02 07:58:29.881199] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 07:58:29.881216] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:01:33.321810] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:01:33.325016] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:01:33.327523] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:01:56.914101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:01:56.914147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:01:56.914218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 07:58:31.290985 (xid=0x1600x) >[2012-10-02 08:01:56.914247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:01:56.914264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:05:23.950135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:05:23.950183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:05:23.950256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:01:58.327462 (xid=0x1601x) >[2012-10-02 08:05:23.950990] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:05:23.951037] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:08:50.982121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:08:50.982166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:08:50.982233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:05:25.360351 (xid=0x1602x) >[2012-10-02 08:08:50.982254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:08:50.982271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:11:33.419431] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:11:33.422874] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:11:33.425582] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:12:18.016129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:12:18.016176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:12:18.016251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:08:52.393566 (xid=0x1603x) >[2012-10-02 08:12:18.017013] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:12:18.017049] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:15:45.050129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:15:45.050179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:15:45.050258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:12:19.427642 (xid=0x1604x) >[2012-10-02 08:15:45.050279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:15:45.050295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:19:12.083114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:19:12.083160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:19:12.083226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:15:46.460403 (xid=0x1605x) >[2012-10-02 08:19:12.083247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:19:12.083264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:21:33.513283] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:21:33.516276] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:21:33.518620] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:22:39.114140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:22:39.114181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:22:39.114244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:19:13.491372 (xid=0x1606x) >[2012-10-02 08:22:39.114264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:22:39.114280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:26:06.146132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:26:06.146182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:26:06.146250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:22:40.523309 (xid=0x1607x) >[2012-10-02 08:26:06.146282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:26:06.146300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:29:33.175126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:29:33.175178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:29:33.175245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:26:07.552759 (xid=0x1608x) >[2012-10-02 08:29:33.175266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:29:33.175282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:31:33.605175] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:31:33.608351] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:31:33.610748] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:33:00.207135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:33:00.207180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:33:00.207246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:29:34.584789 (xid=0x1609x) >[2012-10-02 08:33:00.207268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:33:00.207284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:36:27.242130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:36:27.242177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:36:27.242244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:33:01.619045 (xid=0x1610x) >[2012-10-02 08:36:27.242266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:36:27.242282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:39:54.274099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:39:54.274147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:39:54.274214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:36:28.651022 (xid=0x1611x) >[2012-10-02 08:39:54.274235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:39:54.274252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:41:33.698319] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:41:33.701322] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:41:33.703962] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:43:21.305149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:43:21.305197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:43:21.305265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:39:55.682055 (xid=0x1612x) >[2012-10-02 08:43:21.305286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:43:21.305303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:46:48.337121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:46:48.337168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:46:48.337235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:43:22.714178 (xid=0x1613x) >[2012-10-02 08:46:48.337257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:46:48.337273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:50:15.366136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:50:15.366181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:50:15.366246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:46:49.743703 (xid=0x1614x) >[2012-10-02 08:50:15.366278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:50:15.366295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:51:33.791062] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:51:33.794198] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:51:33.796734] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 08:53:42.397155] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:53:42.397200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:53:42.397266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:50:16.774241 (xid=0x1615x) >[2012-10-02 08:53:42.397287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:53:42.397303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 08:57:09.431106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 08:57:09.431152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 08:57:09.431219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:53:43.808322 (xid=0x1616x) >[2012-10-02 08:57:09.431241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 08:57:09.431258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:00:36.461129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:00:36.461176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:00:36.461244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 08:57:10.838673 (xid=0x1617x) >[2012-10-02 09:00:36.461276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:00:36.461294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:01:33.880625] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:01:33.883820] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:01:33.886651] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:04:03.491136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:04:03.491185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:04:03.491252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:00:37.868713 (xid=0x1618x) >[2012-10-02 09:04:03.491273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:04:03.491290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:07:30.524142] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:07:30.524188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:07:30.524256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:04:04.900854 (xid=0x1619x) >[2012-10-02 09:07:30.524277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:07:30.524294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:10:57.554092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:10:57.554139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:10:57.554207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:07:31.930882 (xid=0x1620x) >[2012-10-02 09:10:57.554228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:10:57.554245] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:11:33.970756] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:11:33.973884] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:11:33.976560] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:14:24.584115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:14:24.584162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:14:24.584227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:10:58.961390 (xid=0x1621x) >[2012-10-02 09:14:24.584248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:14:24.584263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:17:51.618105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:17:51.618151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:17:51.618215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:14:25.995019 (xid=0x1622x) >[2012-10-02 09:17:51.618236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:17:51.618253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:21:18.649116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:21:18.649163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:21:18.649229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:17:53.026521 (xid=0x1623x) >[2012-10-02 09:21:18.649250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:21:18.649266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:21:34.064196] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:21:34.067267] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:21:34.069776] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:24:45.681108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:24:45.681161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:24:45.681228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:21:20.058327 (xid=0x1624x) >[2012-10-02 09:24:45.681249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:24:45.681265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:28:12.714126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:28:12.714174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:28:12.714242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:24:47.091421 (xid=0x1625x) >[2012-10-02 09:28:12.714264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:28:12.714280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:31:34.153515] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:31:34.156360] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:31:34.158883] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:31:39.745112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:31:39.745153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:31:39.745217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:28:14.122274 (xid=0x1626x) >[2012-10-02 09:31:39.745237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:31:39.745266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:35:06.778108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:35:06.778156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:35:06.778223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:31:41.155669 (xid=0x1627x) >[2012-10-02 09:35:06.778245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:35:06.778262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:38:33.810107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:38:33.810154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:38:33.810219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:35:08.187621 (xid=0x1628x) >[2012-10-02 09:38:33.810239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:38:33.810255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:41:34.251251] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:41:34.254818] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:41:34.257917] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:42:00.845111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:42:00.845154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:42:00.845219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:38:35.222554 (xid=0x1629x) >[2012-10-02 09:42:00.845240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:42:00.845256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:45:27.880138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:45:27.880187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:45:27.880265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:42:02.256919 (xid=0x1630x) >[2012-10-02 09:45:27.880286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:45:27.880303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:48:54.912089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:48:54.912140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:48:54.912212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:45:29.289043 (xid=0x1631x) >[2012-10-02 09:48:54.912234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:48:54.912252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:51:34.346676] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:51:34.350174] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:51:34.352890] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 09:52:21.944116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:52:21.944163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:52:21.944229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:48:56.320914 (xid=0x1632x) >[2012-10-02 09:52:21.944250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:52:21.944266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:55:48.976149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:55:48.976198] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:55:48.976263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:52:23.353700 (xid=0x1633x) >[2012-10-02 09:55:48.976295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:55:48.976312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 09:59:16.003033] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 09:59:16.003085] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 09:59:16.003149] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:55:50.380729 (xid=0x1634x) >[2012-10-02 09:59:16.003170] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 09:59:16.003188] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:01:34.428328] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:01:34.431561] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:01:34.433961] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:02:43.031152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:02:43.031199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:02:43.031265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 09:59:17.408064 (xid=0x1635x) >[2012-10-02 10:02:43.031287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:02:43.031303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:06:10.060135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:06:10.060180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:06:10.060245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:02:44.437676 (xid=0x1636x) >[2012-10-02 10:06:10.060267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:06:10.060283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:09:37.089137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:09:37.089190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:09:37.089258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:06:11.466242 (xid=0x1637x) >[2012-10-02 10:09:37.089280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:09:37.089296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:11:34.514274] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:11:34.517797] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:11:34.520388] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:13:04.118139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:13:04.118188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:13:04.118263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:09:38.495178 (xid=0x1638x) >[2012-10-02 10:13:04.118293] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:13:04.118312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:16:31.149135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:16:31.149184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:16:31.149252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:13:05.525899 (xid=0x1639x) >[2012-10-02 10:16:31.149273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:16:31.149290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:19:58.178139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:19:58.178191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:19:58.178278] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:16:32.555043 (xid=0x1640x) >[2012-10-02 10:19:58.178980] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:19:58.179035] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:21:34.600658] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:21:34.603766] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:21:34.606584] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:23:25.207129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:23:25.207177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:23:25.207242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:19:59.584514 (xid=0x1641x) >[2012-10-02 10:23:25.207263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:23:25.207279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:26:52.239107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:26:52.239159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:26:52.239233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:23:26.616133 (xid=0x1642x) >[2012-10-02 10:26:52.239256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:26:52.239273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:30:19.269133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:30:19.269179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:30:19.269243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:26:53.646640 (xid=0x1643x) >[2012-10-02 10:30:19.269275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:30:19.269292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:31:34.689500] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:31:34.692344] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:31:34.694950] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:33:46.299121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:33:46.299170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:33:46.299235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:30:20.676227 (xid=0x1644x) >[2012-10-02 10:33:46.299255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:33:46.299271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:37:13.330087] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:37:13.330132] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:37:13.330197] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:33:47.707296 (xid=0x1645x) >[2012-10-02 10:37:13.330218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:37:13.330233] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:40:40.359104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:40:40.359152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:40:40.359218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:37:14.736490 (xid=0x1646x) >[2012-10-02 10:40:40.359239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:40:40.359256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:41:34.777688] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:41:34.780632] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:41:34.783160] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:44:07.389076] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:44:07.389123] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:44:07.389190] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:40:41.766183 (xid=0x1647x) >[2012-10-02 10:44:07.389212] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:44:07.389228] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:47:34.421091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:47:34.421136] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:47:34.421201] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:44:08.798692 (xid=0x1648x) >[2012-10-02 10:47:34.421222] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:47:34.421238] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:51:01.452110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:51:01.452159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:51:01.452230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:47:35.829020 (xid=0x1649x) >[2012-10-02 10:51:01.452252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:51:01.452270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:51:34.868180] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:51:34.871097] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:51:34.873547] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 10:54:28.482146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:54:28.482198] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:54:28.482267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:51:02.859649 (xid=0x1650x) >[2012-10-02 10:54:28.482289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:54:28.482306] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 10:57:55.514121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 10:57:55.514168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 10:57:55.514236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:54:29.891771 (xid=0x1651x) >[2012-10-02 10:57:55.514258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 10:57:55.514275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:01:22.545099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:01:22.545148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:01:22.545214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 10:57:56.922055 (xid=0x1652x) >[2012-10-02 11:01:22.545235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:01:22.545252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:01:34.958230] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:01:34.961346] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:01:34.963766] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:04:49.575134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:04:49.575182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:04:49.575250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:01:23.952417 (xid=0x1653x) >[2012-10-02 11:04:49.575271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:04:49.575288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:08:16.608139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:08:16.608185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:08:16.608251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:04:50.985768 (xid=0x1654x) >[2012-10-02 11:08:16.608273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:08:16.608290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:11:35.047851] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:11:35.051021] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:11:35.053839] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:11:43.640112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:11:43.640155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:11:43.640219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:08:18.017023 (xid=0x1655x) >[2012-10-02 11:11:43.640240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:11:43.640256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:15:10.674126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:15:10.674171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:15:10.674250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:11:45.051554 (xid=0x1656x) >[2012-10-02 11:15:10.674271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:15:10.674287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:18:37.707116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:18:37.707163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:18:37.707231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:15:12.083858 (xid=0x1657x) >[2012-10-02 11:18:37.707252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:18:37.707269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:21:35.142489] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:21:35.145646] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:21:35.148228] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:22:04.739105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:22:04.739147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:22:04.739212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:18:39.116652 (xid=0x1658x) >[2012-10-02 11:22:04.739232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:22:04.739248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:25:31.771086] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:25:31.771135] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:25:31.771204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:22:06.148019 (xid=0x1659x) >[2012-10-02 11:25:31.771234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:25:31.771252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:28:58.797109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:28:58.797163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:28:58.797238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:25:33.174726 (xid=0x1660x) >[2012-10-02 11:28:58.797259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:28:58.797276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:31:35.223583] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:31:35.226770] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:31:35.229864] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:32:25.824131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:32:25.824172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:32:25.824238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:29:00.201609 (xid=0x1661x) >[2012-10-02 11:32:25.824259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:32:25.824275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:35:52.854143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:35:52.854188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:35:52.854255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:32:27.230997 (xid=0x1662x) >[2012-10-02 11:35:52.854276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:35:52.854304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:39:19.880109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:39:19.880156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:39:19.880222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:35:54.257764 (xid=0x1663x) >[2012-10-02 11:39:19.880244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:39:19.880260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:41:35.309017] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:41:35.312348] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:41:35.315299] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:42:46.908117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:42:46.908161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:42:46.908225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:39:21.285402 (xid=0x1664x) >[2012-10-02 11:42:46.908246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:42:46.908262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:46:13.945093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:46:13.945137] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:46:13.945202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:42:48.322340 (xid=0x1665x) >[2012-10-02 11:46:13.945223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:46:13.945239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:49:40.972116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:49:40.972164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:49:40.972240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:46:15.349284 (xid=0x1666x) >[2012-10-02 11:49:40.972262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:49:40.972278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:51:35.393403] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:51:35.396268] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:51:35.398535] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 11:53:07.999139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:53:07.999188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:53:07.999262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:49:42.376395 (xid=0x1667x) >[2012-10-02 11:53:07.999292] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:53:07.999311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 11:56:35.029071] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 11:56:35.029121] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 11:56:35.029196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:53:09.406051 (xid=0x1668x) >[2012-10-02 11:56:35.030255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 11:56:35.030298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:00:02.056101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:00:02.056152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:00:02.056217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 11:56:36.433169 (xid=0x1669x) >[2012-10-02 12:00:02.056249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:00:02.056267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:01:35.475857] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:01:35.479282] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:01:35.482152] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:03:29.083122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:03:29.083172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:03:29.083240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:00:03.460320 (xid=0x1670x) >[2012-10-02 12:03:29.083261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:03:29.083278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:06:56.113133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:06:56.113183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:06:56.113248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:03:30.490623 (xid=0x1671x) >[2012-10-02 12:06:56.113269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:06:56.113286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:10:23.141119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:10:23.141167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:10:23.141234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:06:57.518263 (xid=0x1672x) >[2012-10-02 12:10:23.141256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:10:23.141272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:11:35.559487] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:11:35.562428] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:11:35.564870] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:13:50.169149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:13:50.169197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:13:50.169264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:10:24.546051 (xid=0x1673x) >[2012-10-02 12:13:50.169285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:13:50.169302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:17:17.200130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:17:17.200176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:17:17.200243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:13:51.577666 (xid=0x1674x) >[2012-10-02 12:17:17.200265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:17:17.200283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:20:44.228123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:20:44.228173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:20:44.228240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:17:18.605518 (xid=0x1675x) >[2012-10-02 12:20:44.228261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:20:44.228278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:21:35.646572] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:21:35.649991] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:21:35.653091] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:24:11.258108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:24:11.258156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:24:11.258223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:20:45.635066 (xid=0x1676x) >[2012-10-02 12:24:11.258244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:24:11.258260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:27:38.288145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:27:38.288190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:27:38.288255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:24:12.665303 (xid=0x1677x) >[2012-10-02 12:27:38.288276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:27:38.288293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:31:05.315130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:31:05.315179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:31:05.315251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:27:39.692597 (xid=0x1678x) >[2012-10-02 12:31:05.315272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:31:05.315289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:31:35.728204] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:31:35.731577] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:31:35.734550] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:34:32.343120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:34:32.343166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:34:32.343231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:31:06.720362 (xid=0x1679x) >[2012-10-02 12:34:32.343252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:34:32.343268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:37:59.374100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:37:59.374150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:37:59.374218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:34:33.750893 (xid=0x1680x) >[2012-10-02 12:37:59.374240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:37:59.374257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:41:26.401105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:41:26.401152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:41:26.401219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:38:00.778573 (xid=0x1681x) >[2012-10-02 12:41:26.401241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:41:26.401258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:41:35.812912] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:41:35.815777] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:41:35.818153] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:44:53.430119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:44:53.430180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:44:53.430246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:41:27.807076 (xid=0x1682x) >[2012-10-02 12:44:53.430266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:44:53.430282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:48:20.459122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:48:20.459171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:48:20.459238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:44:54.836065 (xid=0x1683x) >[2012-10-02 12:48:20.459259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:48:20.459276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:51:35.889695] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:51:35.891912] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:51:35.894028] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 12:51:47.487128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:51:47.487172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:51:47.487237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:48:21.863998 (xid=0x1684x) >[2012-10-02 12:51:47.487258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:51:47.487273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:55:14.515118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:55:14.515169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:55:14.515237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:51:48.892304 (xid=0x1685x) >[2012-10-02 12:55:14.515269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:55:14.515286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 12:58:41.542109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 12:58:41.542155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 12:58:41.542223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:55:15.919486 (xid=0x1686x) >[2012-10-02 12:58:41.542245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 12:58:41.542261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:01:35.975562] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:01:35.978415] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:01:35.980932] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:02:08.575120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:02:08.575172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:02:08.575245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 12:58:42.952130 (xid=0x1687x) >[2012-10-02 13:02:08.575268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:02:08.575285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:05:35.606118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:05:35.606164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:05:35.606231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:02:09.983085 (xid=0x1688x) >[2012-10-02 13:05:35.606253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:05:35.606280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:09:02.633111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:09:02.633165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:09:02.633234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:05:37.010506 (xid=0x1689x) >[2012-10-02 13:09:02.633256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:09:02.633272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:11:36.060865] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:11:36.064093] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:11:36.066648] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:12:29.661101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:12:29.661147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:12:29.661218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:09:04.038328 (xid=0x1690x) >[2012-10-02 13:12:29.661239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:12:29.661256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:15:56.691094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:15:56.691145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:15:56.691222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:12:31.068107 (xid=0x1691x) >[2012-10-02 13:15:56.691245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:15:56.691262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:19:23.719111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:19:23.719166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:19:23.719241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:15:58.096435 (xid=0x1692x) >[2012-10-02 13:19:23.719261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:19:23.719277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:21:36.144584] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:21:36.147316] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:21:36.149406] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:22:50.748024] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:22:50.748069] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:22:50.748133] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:19:25.124924 (xid=0x1693x) >[2012-10-02 13:22:50.748155] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:22:50.748170] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:26:17.778107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:26:17.778153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:26:17.778220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:22:52.155362 (xid=0x1694x) >[2012-10-02 13:26:17.778242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:26:17.778259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:29:44.807110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:29:44.807155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:29:44.807222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:26:19.184022 (xid=0x1695x) >[2012-10-02 13:29:44.807255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:29:44.807272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:31:36.230138] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:31:36.233256] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:31:36.236049] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:33:11.835097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:33:11.835142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:33:11.835207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:29:46.212339 (xid=0x1696x) >[2012-10-02 13:33:11.835227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:33:11.835242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:36:38.865116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:36:38.865162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:36:38.865228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:33:13.242615 (xid=0x1697x) >[2012-10-02 13:36:38.865250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:36:38.865267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:40:05.894096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:40:05.894143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:40:05.894212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:36:40.271392 (xid=0x1698x) >[2012-10-02 13:40:05.894234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:40:05.894262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:41:36.316061] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:41:36.319276] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:41:36.322213] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:43:32.923095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:43:32.923140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:43:32.923204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:40:07.300616 (xid=0x1699x) >[2012-10-02 13:43:32.923225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:43:32.923242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:46:59.954097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:46:59.954145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:46:59.954212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:43:34.331192 (xid=0x1700x) >[2012-10-02 13:46:59.954234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:46:59.954251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:50:26.983139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:50:26.983185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:50:26.983250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:47:01.360098 (xid=0x1701x) >[2012-10-02 13:50:26.983271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:50:26.983288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:51:36.401826] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:51:36.404969] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:51:36.407626] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 13:53:54.012162] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:53:54.012208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:53:54.012275] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:50:28.389433 (xid=0x1702x) >[2012-10-02 13:53:54.012296] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:53:54.012313] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 13:57:21.043089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 13:57:21.043140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 13:57:21.043208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:53:55.420665 (xid=0x1703x) >[2012-10-02 13:57:21.043229] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 13:57:21.043246] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:00:48.074127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:00:48.074173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:00:48.074241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 13:57:22.451693 (xid=0x1704x) >[2012-10-02 14:00:48.074263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:00:48.074279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:01:36.494440] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:01:36.497506] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:01:36.499857] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:04:15.105104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:04:15.105157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:04:15.105223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:00:49.482282 (xid=0x1705x) >[2012-10-02 14:04:15.105244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:04:15.105261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:07:42.137132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:07:42.137179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:07:42.137253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:04:16.513845 (xid=0x1706x) >[2012-10-02 14:07:42.137282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:07:42.137300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:11:09.166125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:11:09.166171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:11:09.166238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:07:43.543786 (xid=0x1707x) >[2012-10-02 14:11:09.166259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:11:09.166275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:11:36.582439] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:11:36.585697] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:11:36.588133] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:14:36.197122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:14:36.197180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:14:36.197247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:11:10.574022 (xid=0x1708x) >[2012-10-02 14:14:36.197267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:14:36.197284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:18:03.230149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:18:03.230195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:18:03.230263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:14:37.607587 (xid=0x1709x) >[2012-10-02 14:18:03.230285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:18:03.230302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:21:30.261119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:21:30.261166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:21:30.261232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:18:04.638816 (xid=0x1710x) >[2012-10-02 14:21:30.261254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:21:30.261271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:21:36.675775] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:21:36.679150] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:21:36.682739] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:24:57.293140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:24:57.293189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:24:57.293264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:21:31.670414 (xid=0x1711x) >[2012-10-02 14:24:57.294114] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:24:57.294150] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:28:24.325137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:28:24.325188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:28:24.325263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:24:58.702363 (xid=0x1712x) >[2012-10-02 14:28:24.326000] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:28:24.326050] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:31:36.760461] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:31:36.763416] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:31:36.766029] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:31:51.355120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:31:51.355162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:31:51.355225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:28:25.732113 (xid=0x1713x) >[2012-10-02 14:31:51.355246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:31:51.355261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:35:18.387140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:35:18.387188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:35:18.387253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:31:52.764369 (xid=0x1714x) >[2012-10-02 14:35:18.387285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:35:18.387303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:38:45.417121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:38:45.417168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:38:45.417234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:35:19.794247 (xid=0x1715x) >[2012-10-02 14:38:45.417255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:38:45.417271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:41:36.853337] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:41:36.856483] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:41:36.858967] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:42:12.447110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:42:12.447157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:42:12.447229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:38:46.824578 (xid=0x1716x) >[2012-10-02 14:42:12.447251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:42:12.447268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:45:39.483112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:45:39.483158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:45:39.483224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:42:13.860022 (xid=0x1717x) >[2012-10-02 14:45:39.483246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:45:39.483263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:49:06.514098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:49:06.514155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:49:06.514224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:45:40.891597 (xid=0x1718x) >[2012-10-02 14:49:06.514245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:49:06.514261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:51:36.945400] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:51:36.948264] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:51:36.951042] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 14:52:33.545105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:52:33.545150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:52:33.545215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:49:07.922421 (xid=0x1719x) >[2012-10-02 14:52:33.545237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:52:33.545253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:56:00.577144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:56:00.577192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:56:00.577259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:52:34.954747 (xid=0x1720x) >[2012-10-02 14:56:00.577280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:56:00.577296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 14:59:27.608085] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 14:59:27.608132] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 14:59:27.608198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:56:01.985517 (xid=0x1721x) >[2012-10-02 14:59:27.608231] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 14:59:27.608247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:01:37.037027] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:01:37.039833] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:01:37.042153] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:02:54.639135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:02:54.639183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:02:54.639249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 14:59:29.016554 (xid=0x1722x) >[2012-10-02 15:02:54.639271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:02:54.639287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:06:21.673091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:06:21.673138] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:06:21.673205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:02:56.049840 (xid=0x1723x) >[2012-10-02 15:06:21.673228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:06:21.673244] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:09:48.704098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:09:48.704145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:09:48.704211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:06:23.081412 (xid=0x1724x) >[2012-10-02 15:09:48.704234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:09:48.704261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:11:37.132755] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:11:37.136262] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:11:37.139434] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:13:15.736092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:13:15.736140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:13:15.736205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:09:50.113297 (xid=0x1725x) >[2012-10-02 15:13:15.736227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:13:15.736243] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:16:42.771107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:16:42.771151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:16:42.771216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:13:17.148132 (xid=0x1726x) >[2012-10-02 15:16:42.771237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:16:42.771253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:20:09.802154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:20:09.802200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:20:09.802274] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:16:44.179034 (xid=0x1727x) >[2012-10-02 15:20:09.802296] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:20:09.802313] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:21:37.227671] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:21:37.230779] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:21:37.233822] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:23:36.834102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:23:36.834150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:23:36.834217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:20:11.211775 (xid=0x1728x) >[2012-10-02 15:23:36.834238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:23:36.834254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:27:03.870032] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:27:03.870085] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:27:03.870149] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:23:38.246412 (xid=0x1729x) >[2012-10-02 15:27:03.870170] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:27:03.870186] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:30:30.901106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:30:30.901154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:30:30.901231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:27:05.278178 (xid=0x1730x) >[2012-10-02 15:30:30.902194] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:30:30.902237] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:31:37.323440] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:31:37.326374] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:31:37.329326] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:33:57.932116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:33:57.932165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:33:57.932230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:30:32.309599 (xid=0x1731x) >[2012-10-02 15:33:57.932252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:33:57.932269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:37:24.967153] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:37:24.967200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:37:24.967264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:33:59.344055 (xid=0x1732x) >[2012-10-02 15:37:24.967286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:37:24.967303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:40:51.997104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:40:51.997152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:40:51.997219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:37:26.374221 (xid=0x1733x) >[2012-10-02 15:40:51.997241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:40:51.997258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:41:37.411961] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:41:37.414789] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:41:37.417120] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:44:19.025120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:44:19.025168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:44:19.025235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:40:53.402269 (xid=0x1734x) >[2012-10-02 15:44:19.025255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:44:19.025271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:47:46.054123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:47:46.054169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:47:46.054235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:44:20.431138 (xid=0x1735x) >[2012-10-02 15:47:46.054256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:47:46.054273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:51:13.082134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:51:13.082183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:51:13.082251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:47:47.459395 (xid=0x1736x) >[2012-10-02 15:51:13.082272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:51:13.082289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:51:37.495785] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:51:37.498662] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:51:37.500973] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 15:54:40.111065] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:54:40.111113] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:54:40.111191] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:51:14.488584 (xid=0x1737x) >[2012-10-02 15:54:40.111212] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:54:40.111230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 15:58:07.141136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 15:58:07.141184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 15:58:07.141258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:54:41.517839 (xid=0x1738x) >[2012-10-02 15:58:07.141280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 15:58:07.141298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:01:34.170107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:01:34.170154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:01:34.170221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 15:58:08.547065 (xid=0x1739x) >[2012-10-02 16:01:34.170243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:01:34.170260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:01:37.581739] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:01:37.584793] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:01:37.587392] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:05:01.199146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:05:01.199192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:05:01.199259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:01:35.576763 (xid=0x1740x) >[2012-10-02 16:05:01.199291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:05:01.199308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:08:28.231146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:08:28.231193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:08:28.231258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:05:02.608654 (xid=0x1741x) >[2012-10-02 16:08:28.231280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:08:28.231297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:11:37.667392] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:11:37.670590] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:11:37.673471] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:11:55.262123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:11:55.262167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:11:55.262230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:08:29.639159 (xid=0x1742x) >[2012-10-02 16:11:55.262251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:11:55.262267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:15:22.293140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:15:22.293188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:15:22.293253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:11:56.670472 (xid=0x1743x) >[2012-10-02 16:15:22.293274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:15:22.293290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:18:49.322108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:18:49.322154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:18:49.322220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:15:23.699135 (xid=0x1744x) >[2012-10-02 16:18:49.322241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:18:49.322258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:21:37.753946] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:21:37.757237] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:21:37.759990] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:22:16.352110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:22:16.352157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:22:16.352222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:18:50.728823 (xid=0x1745x) >[2012-10-02 16:22:16.352243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:22:16.352258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:25:43.382105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:25:43.382150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:25:43.382212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:22:17.759683 (xid=0x1746x) >[2012-10-02 16:25:43.382233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:25:43.382249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:29:10.412122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:29:10.412167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:29:10.412235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:25:44.789510 (xid=0x1747x) >[2012-10-02 16:29:10.412266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:29:10.412283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:31:37.843171] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:31:37.846443] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:31:37.849437] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:32:37.442105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:32:37.442148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:32:37.442212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:29:11.819781 (xid=0x1748x) >[2012-10-02 16:32:37.442233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:32:37.442248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:36:04.475119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:36:04.475164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:36:04.475232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:32:38.852195 (xid=0x1749x) >[2012-10-02 16:36:04.475253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:36:04.475269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:39:31.506118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:39:31.506166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:39:31.506232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:36:05.883022 (xid=0x1750x) >[2012-10-02 16:39:31.506271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:39:31.506289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:41:37.938013] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:41:37.940761] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:41:37.942817] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:42:58.540122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:42:58.540164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:42:58.540228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:39:32.917539 (xid=0x1751x) >[2012-10-02 16:42:58.540249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:42:58.540265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:46:25.572111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:46:25.572155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:46:25.572220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:42:59.949555 (xid=0x1752x) >[2012-10-02 16:46:25.572242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:46:25.572257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:49:52.602143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:49:52.602189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:49:52.602255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:46:26.979240 (xid=0x1753x) >[2012-10-02 16:49:52.602277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:49:52.602294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:51:38.027958] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:51:38.031676] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:51:38.034665] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 16:53:19.632141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:53:19.632187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:53:19.632253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:49:54.009201 (xid=0x1754x) >[2012-10-02 16:53:19.632273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:53:19.632290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 16:56:46.670639] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 16:56:46.670693] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 16:56:46.670775] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:53:21.043144 (xid=0x1755x) >[2012-10-02 16:56:46.670797] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 16:56:46.670823] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:00:13.697110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:00:13.697158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:00:13.697233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 16:56:48.074670 (xid=0x1756x) >[2012-10-02 17:00:13.697255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:00:13.697273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:01:38.124008] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:01:38.127619] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:01:38.130727] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:03:40.730001] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:03:40.730077] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:03:40.730159] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:00:15.108044 (xid=0x1757x) >[2012-10-02 17:03:40.730182] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:03:40.730198] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:07:10.763121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:07:10.763170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:07:10.763240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:03:45.140135 (xid=0x1758x) >[2012-10-02 17:07:10.763261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:07:10.763278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:10:37.795120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:10:37.795171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:10:37.795247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:07:12.171841 (xid=0x1759x) >[2012-10-02 17:10:37.796428] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:10:37.796464] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:11:38.215812] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:11:38.219220] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:11:38.222505] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:14:04.826104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:14:04.826159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:14:04.826234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:10:39.203376 (xid=0x1760x) >[2012-10-02 17:14:04.826256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:14:04.826274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:17:31.860107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:17:31.860154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:17:31.860221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:14:06.237671 (xid=0x1761x) >[2012-10-02 17:17:31.860243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:17:31.860259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:20:58.892112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:20:58.892159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:20:58.892227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:17:33.269277 (xid=0x1762x) >[2012-10-02 17:20:58.892248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:20:58.892265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:21:38.310889] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:21:38.314429] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:21:38.317580] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:24:25.924112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:24:25.924159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:24:25.924239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:21:00.301651 (xid=0x1763x) >[2012-10-02 17:24:25.924260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:24:25.924276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:27:52.955115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:27:52.955168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:27:52.955241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:24:27.332303 (xid=0x1764x) >[2012-10-02 17:27:52.956186] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:27:52.956230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:31:19.986128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:31:19.986177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:31:19.986244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:27:54.363554 (xid=0x1765x) >[2012-10-02 17:31:19.986265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:31:19.986282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:31:38.396785] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:31:38.400138] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:31:38.403308] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:34:47.013103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:34:47.013154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:34:47.013222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:31:21.390634 (xid=0x1766x) >[2012-10-02 17:34:47.013254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:34:47.013271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:38:14.043115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:38:14.043166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:38:14.043240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:34:48.419880 (xid=0x1767x) >[2012-10-02 17:38:14.043263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:38:14.043281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:41:38.479841] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:41:38.482973] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:41:38.485191] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:41:41.073100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:41:41.073139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:41:41.073204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:38:15.450711 (xid=0x1768x) >[2012-10-02 17:41:41.073225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:41:41.073241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:45:08.104112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:45:08.104160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:45:08.104235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:41:42.481008 (xid=0x1769x) >[2012-10-02 17:45:08.104258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:45:08.104276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:48:35.133086] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:48:35.133136] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:48:35.133204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:45:09.509982 (xid=0x1770x) >[2012-10-02 17:48:35.133226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:48:35.133243] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:51:38.567283] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:51:38.570594] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:51:38.573276] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 17:52:02.163112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:52:02.163153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:52:02.163216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:48:36.540847 (xid=0x1771x) >[2012-10-02 17:52:02.163237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:52:02.163253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:55:29.195112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:55:29.195164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:55:29.195235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:52:03.572163 (xid=0x1772x) >[2012-10-02 17:55:29.196179] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:55:29.196223] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 17:58:56.225117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 17:58:56.225163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 17:58:56.225242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:55:30.602155 (xid=0x1773x) >[2012-10-02 17:58:56.225263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 17:58:56.225280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:01:38.655800] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:01:38.658921] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:01:38.661349] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:02:23.253117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:02:23.253160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:02:23.253225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 17:58:57.630075 (xid=0x1774x) >[2012-10-02 18:02:23.253246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:02:23.253262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:05:50.287125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:05:50.287172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:05:50.287237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:02:24.664822 (xid=0x1775x) >[2012-10-02 18:05:50.287258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:05:50.287275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:09:17.315127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:09:17.315174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:09:17.315249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:05:51.692921 (xid=0x1776x) >[2012-10-02 18:09:17.315283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:09:17.315301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:11:38.745094] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:11:38.748150] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:11:38.750824] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:12:44.345020] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:12:44.345066] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:12:44.345131] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:09:18.722018 (xid=0x1777x) >[2012-10-02 18:12:44.345152] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:12:44.345167] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:16:11.378100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:16:11.378150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:16:11.378227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:12:45.755519 (xid=0x1778x) >[2012-10-02 18:16:11.379185] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:16:11.379228] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:19:38.408138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:19:38.408185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:19:38.408252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:16:12.785287 (xid=0x1779x) >[2012-10-02 18:19:38.408274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:19:38.408291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:21:38.834771] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:21:38.837898] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:21:38.840387] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:23:05.438077] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:23:05.438125] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:23:05.438192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:19:39.815549 (xid=0x1780x) >[2012-10-02 18:23:05.438213] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:23:05.438230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:26:32.471132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:26:32.471181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:26:32.471248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:23:06.848177 (xid=0x1781x) >[2012-10-02 18:26:32.471269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:26:32.471287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:29:59.502113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:29:59.502160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:29:59.502225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:26:33.879150 (xid=0x1782x) >[2012-10-02 18:29:59.502246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:29:59.502262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:31:38.940172] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:31:38.943619] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:31:38.946606] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:33:26.548112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:33:26.548162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:33:26.548227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:30:00.924970 (xid=0x1783x) >[2012-10-02 18:33:26.548248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:33:26.548263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:36:53.577121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:36:53.577170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:36:53.577237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:33:27.954782 (xid=0x1784x) >[2012-10-02 18:36:53.577259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:36:53.577276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:40:20.605114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:40:20.605160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:40:20.605228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:36:54.982465 (xid=0x1785x) >[2012-10-02 18:40:20.605249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:40:20.605266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:41:39.022401] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:41:39.025935] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:41:39.028317] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:43:47.632127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:43:47.632176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:43:47.632245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:40:22.009336 (xid=0x1786x) >[2012-10-02 18:43:47.632266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:43:47.632283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:47:14.662153] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:47:14.662211] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:47:14.662286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:43:49.038899 (xid=0x1787x) >[2012-10-02 18:47:14.662308] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:47:14.662326] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:50:41.690114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:50:41.690165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:50:41.690233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:47:16.067475 (xid=0x1788x) >[2012-10-02 18:50:41.690255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:50:41.690271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:51:39.112789] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:51:39.116263] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:51:39.119292] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 18:54:08.723133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:54:08.723186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:54:08.723265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:50:43.099883 (xid=0x1789x) >[2012-10-02 18:54:08.723286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:54:08.723302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 18:57:35.758128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 18:57:35.758179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 18:57:35.758247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:54:10.135184 (xid=0x1790x) >[2012-10-02 18:57:35.758269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 18:57:35.758285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:01:02.791152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:01:02.791209] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:01:02.791285] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 18:57:37.168892 (xid=0x1791x) >[2012-10-02 19:01:02.791306] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:01:02.791323] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:01:39.210327] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:01:39.213486] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:01:39.216159] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:04:29.824152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:04:29.824201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:04:29.824269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:01:04.201648 (xid=0x1792x) >[2012-10-02 19:04:29.824302] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:04:29.824320] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:07:56.858079] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:07:56.858126] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:07:56.858192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:04:31.235499 (xid=0x1793x) >[2012-10-02 19:07:56.858214] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:07:56.858230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:11:23.886126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:11:23.886174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:11:23.886241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:07:58.263411 (xid=0x1794x) >[2012-10-02 19:11:23.886262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:11:23.886279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:11:39.300138] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:11:39.302396] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:11:39.304440] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:14:50.916092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:14:50.916139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:14:50.916207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:11:25.294030 (xid=0x1795x) >[2012-10-02 19:14:50.916228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:14:50.916255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:18:17.952077] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:18:17.952125] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:18:17.952192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:14:52.329492 (xid=0x1796x) >[2012-10-02 19:18:17.952213] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:18:17.952229] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:21:39.390254] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:21:39.393334] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:21:39.396377] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:21:44.983131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:21:44.983172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:21:44.983237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:18:19.360366 (xid=0x1797x) >[2012-10-02 19:21:44.983264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:21:44.983281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:25:12.015084] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:25:12.015131] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:25:12.015201] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:21:46.392394 (xid=0x1798x) >[2012-10-02 19:25:12.015222] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:25:12.015239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:28:39.042131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:28:39.042180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:28:39.042261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:25:13.419567 (xid=0x1799x) >[2012-10-02 19:28:39.042282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:28:39.042298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:31:39.474861] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:31:39.477972] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:31:39.480368] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:32:06.069106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:32:06.069151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:32:06.069213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:28:40.446971 (xid=0x1800x) >[2012-10-02 19:32:06.069234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:32:06.069250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:35:33.102110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:35:33.102160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:35:33.102235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:32:07.479347 (xid=0x1801x) >[2012-10-02 19:35:33.103271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:35:33.103316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:39:00.129124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:39:00.129171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:39:00.129238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:35:34.506865 (xid=0x1802x) >[2012-10-02 19:39:00.129271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:39:00.129287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:41:39.560967] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:41:39.564262] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:41:39.566667] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:42:27.159121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:42:27.159171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:42:27.159235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:39:01.536990 (xid=0x1803x) >[2012-10-02 19:42:27.159256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:42:27.159273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:45:54.192099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:45:54.192149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:45:54.192225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:42:28.569169 (xid=0x1804x) >[2012-10-02 19:45:54.193653] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:45:54.193689] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:49:21.222133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:49:21.222185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:49:21.222251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:45:55.599548 (xid=0x1805x) >[2012-10-02 19:49:21.222273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:49:21.222289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:51:39.652954] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:51:39.655945] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:51:39.658566] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 19:52:48.253141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:52:48.253190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:52:48.253255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:49:22.630643 (xid=0x1806x) >[2012-10-02 19:52:48.253275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:52:48.253292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:56:15.286101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:56:15.286151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:56:15.286226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:52:49.662819 (xid=0x1807x) >[2012-10-02 19:56:15.287289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:56:15.287333] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 19:59:42.313129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 19:59:42.313178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 19:59:42.313244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:56:16.690660 (xid=0x1808x) >[2012-10-02 19:59:42.313265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 19:59:42.313281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:01:39.738358] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:01:39.741792] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:01:39.744934] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:03:09.342110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:03:09.342161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:03:09.342236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 19:59:43.719757 (xid=0x1809x) >[2012-10-02 20:03:09.343005] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:03:09.343041] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:06:36.373116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:06:36.373166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:06:36.373234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:03:10.750208 (xid=0x1810x) >[2012-10-02 20:06:36.373256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:06:36.373272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:10:03.402069] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:10:03.402118] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:10:03.402188] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:06:37.779081 (xid=0x1811x) >[2012-10-02 20:10:03.402209] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:10:03.402226] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:11:39.825578] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:11:39.828115] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:11:39.830158] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:13:30.432099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:13:30.432150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:13:30.432217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:10:04.809013 (xid=0x1812x) >[2012-10-02 20:13:30.432238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:13:30.432253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:16:57.462101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:16:57.462152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:16:57.462219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:13:31.839329 (xid=0x1813x) >[2012-10-02 20:16:57.462240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:16:57.462255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:20:24.492128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:20:24.492175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:20:24.492241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:16:58.869555 (xid=0x1814x) >[2012-10-02 20:20:24.492262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:20:24.492278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:21:39.913726] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:21:39.916873] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:21:39.919722] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:23:51.523130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:23:51.523190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:23:51.523257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:20:25.899973 (xid=0x1815x) >[2012-10-02 20:23:51.523277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:23:51.523292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:27:18.555039] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:27:18.555089] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:27:18.555157] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:23:52.932026 (xid=0x1816x) >[2012-10-02 20:27:18.555179] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:27:18.555195] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:30:48.585075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:30:48.585126] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:30:48.585193] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:27:22.962161 (xid=0x1817x) >[2012-10-02 20:30:48.585215] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:30:48.585231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:31:40.005284] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:31:40.008539] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:31:40.011055] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:34:15.615116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:34:15.615169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:34:15.615238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:30:49.992773 (xid=0x1818x) >[2012-10-02 20:34:15.615273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:34:15.615290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:37:42.650125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:37:42.650178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:37:42.650245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:34:17.027822 (xid=0x1819x) >[2012-10-02 20:37:42.650267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:37:42.650283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:41:09.682119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:41:09.682171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:41:09.682240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:37:44.059402 (xid=0x1820x) >[2012-10-02 20:41:09.682261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:41:09.682277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:41:40.100011] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:41:40.102725] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:41:40.104891] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:44:36.714113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:44:36.714165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:44:36.714231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:41:11.091848 (xid=0x1821x) >[2012-10-02 20:44:36.714252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:44:36.714280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:48:03.750101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:48:03.750152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:48:03.750218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:44:38.127072 (xid=0x1822x) >[2012-10-02 20:48:03.750238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:48:03.750255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:51:30.782124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:51:30.782177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:51:30.782246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:48:05.159533 (xid=0x1823x) >[2012-10-02 20:51:30.782267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:51:30.782284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:51:40.197819] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:51:40.200721] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:51:40.203035] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 20:54:57.815102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:54:57.815155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:54:57.815223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:51:32.192182 (xid=0x1824x) >[2012-10-02 20:54:57.815244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:54:57.815260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 20:58:24.850122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 20:58:24.850183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 20:58:24.850251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:54:59.227163 (xid=0x1825x) >[2012-10-02 20:58:24.850271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 20:58:24.850287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:01:40.291982] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:01:40.295419] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:01:40.297905] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:01:51.882108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:01:51.882153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:01:51.882218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 20:58:26.259818 (xid=0x1826x) >[2012-10-02 21:01:51.882238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:01:51.882254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:05:18.919124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:05:18.919179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:05:18.919254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:01:53.296637 (xid=0x1827x) >[2012-10-02 21:05:18.920864] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:05:18.920901] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:08:45.954115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:08:45.954167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:08:45.954234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:05:20.331239 (xid=0x1828x) >[2012-10-02 21:08:45.954269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:08:45.954287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:11:40.385744] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:11:40.388852] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:11:40.391227] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:12:12.985107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:12:12.985156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:12:12.985222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:08:47.362517 (xid=0x1829x) >[2012-10-02 21:12:12.985243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:12:12.985259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:15:40.015133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:15:40.015186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:15:40.015253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:12:14.392285 (xid=0x1830x) >[2012-10-02 21:15:40.015275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:15:40.015292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:19:07.044119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:19:07.044169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:19:07.044234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:15:41.421664 (xid=0x1831x) >[2012-10-02 21:19:07.044256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:19:07.044282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:21:40.474586] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:21:40.477793] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:21:40.480247] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:22:34.074116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:22:34.074164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:22:34.074228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:19:08.451660 (xid=0x1832x) >[2012-10-02 21:22:34.074248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:22:34.074264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:26:01.105131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:26:01.105181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:26:01.105248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:22:35.482696 (xid=0x1833x) >[2012-10-02 21:26:01.105269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:26:01.105286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:29:28.133116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:29:28.133168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:29:28.133237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:26:02.510298 (xid=0x1834x) >[2012-10-02 21:29:28.133258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:29:28.133275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:31:40.561350] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:31:40.564429] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:31:40.567020] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:32:55.163114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:32:55.163167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:32:55.163234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:29:29.540586 (xid=0x1835x) >[2012-10-02 21:32:55.163262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:32:55.163280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:36:22.196143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:36:22.196200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:36:22.196275] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:32:56.573440 (xid=0x1836x) >[2012-10-02 21:36:22.196296] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:36:22.196314] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:39:49.224117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:39:49.224173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:39:49.224238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:36:23.601057 (xid=0x1837x) >[2012-10-02 21:39:49.224259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:39:49.224276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:41:40.651537] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:41:40.654832] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:41:40.657850] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:43:16.255102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:43:16.255155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:43:16.255225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:39:50.632405 (xid=0x1838x) >[2012-10-02 21:43:16.255246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:43:16.255263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:46:43.289119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:46:43.289170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:46:43.289235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:43:17.665854 (xid=0x1839x) >[2012-10-02 21:46:43.289257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:46:43.289273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:50:10.319123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:50:10.319171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:50:10.319239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:46:44.696668 (xid=0x1840x) >[2012-10-02 21:50:10.319260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:50:10.319276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:51:40.740184] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:51:40.742717] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:51:40.744889] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 21:53:37.347090] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:53:37.347140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:53:37.347205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:50:11.725021 (xid=0x1841x) >[2012-10-02 21:53:37.347226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:53:37.347242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 21:57:04.381123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 21:57:04.381174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 21:57:04.381239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:53:38.758520 (xid=0x1842x) >[2012-10-02 21:57:04.381261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 21:57:04.381277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:00:31.411069] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:00:31.411121] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:00:31.411189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 21:57:05.788627 (xid=0x1843x) >[2012-10-02 22:00:31.411211] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:00:31.411227] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:01:40.834011] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:01:40.836760] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:01:40.838830] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:03:58.442127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:03:58.442178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:03:58.442244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:00:32.819185 (xid=0x1844x) >[2012-10-02 22:03:58.442277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:03:58.442294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:07:25.477116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:07:25.477180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:07:25.477259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:03:59.854691 (xid=0x1845x) >[2012-10-02 22:07:25.477281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:07:25.477298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:10:52.508146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:10:52.508197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:10:52.508261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:07:26.885207 (xid=0x1846x) >[2012-10-02 22:10:52.508282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:10:52.508298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:11:40.925360] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:11:40.928921] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:11:40.931810] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:14:19.538117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:14:19.538170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:14:19.538244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:10:53.914852 (xid=0x1847x) >[2012-10-02 22:14:19.539086] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:14:19.539119] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:17:46.569102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:17:46.569153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:17:46.569222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:14:20.947013 (xid=0x1848x) >[2012-10-02 22:17:46.569243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:17:46.569259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:21:13.601124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:21:13.601173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:21:13.601240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:17:47.978744 (xid=0x1849x) >[2012-10-02 22:21:13.601261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:21:13.601277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:21:41.018762] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:21:41.021790] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:21:41.024097] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:24:40.634114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:24:40.634165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:24:40.634231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:21:15.011064 (xid=0x1850x) >[2012-10-02 22:24:40.634252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:24:40.634268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:28:07.665120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:28:07.665181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:28:07.665248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:24:42.042046 (xid=0x1851x) >[2012-10-02 22:28:07.665269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:28:07.665285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:31:34.694094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:31:34.694146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:31:34.694215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:28:09.071220 (xid=0x1852x) >[2012-10-02 22:31:34.694236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:31:34.694253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:31:41.105878] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:31:41.108930] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:31:41.111373] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:35:01.723122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:35:01.723174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:35:01.723241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:31:36.100717 (xid=0x1853x) >[2012-10-02 22:35:01.723262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:35:01.723279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:38:28.756134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:38:28.756186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:38:28.756256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:35:03.133216 (xid=0x1854x) >[2012-10-02 22:38:28.756289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:38:28.756306] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:41:41.194016] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:41:41.197562] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:41:41.200400] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:41:55.787116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:41:55.787161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:41:55.787226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:38:30.164100 (xid=0x1855x) >[2012-10-02 22:41:55.787247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:41:55.787263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:45:22.820116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:45:22.820167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:45:22.820234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:41:57.197190 (xid=0x1856x) >[2012-10-02 22:45:22.820255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:45:22.820271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:48:49.851110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:48:49.851161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:48:49.851227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:45:24.228183 (xid=0x1857x) >[2012-10-02 22:48:49.851249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:48:49.851278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:51:41.288310] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:51:41.291604] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:51:41.294061] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 22:52:16.884103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:52:16.884167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:52:16.884245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:48:51.261814 (xid=0x1858x) >[2012-10-02 22:52:16.884267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:52:16.884284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:55:43.917097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:55:43.917149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:55:43.917216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:52:18.293912 (xid=0x1859x) >[2012-10-02 22:55:43.917237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:55:43.917254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 22:59:10.947101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 22:59:10.947150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 22:59:10.947217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:55:45.324066 (xid=0x1860x) >[2012-10-02 22:59:10.947238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 22:59:10.947255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:01:41.376110] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:01:41.378659] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:01:41.380746] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:02:37.977111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:02:37.977161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:02:37.977236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 22:59:12.354056 (xid=0x1861x) >[2012-10-02 23:02:37.978047] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:02:37.978093] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:06:05.007130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:06:05.007181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:06:05.007249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:02:39.384050 (xid=0x1862x) >[2012-10-02 23:06:05.007270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:06:05.007287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:09:32.035098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:09:32.035152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:09:32.035221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:06:06.412264 (xid=0x1863x) >[2012-10-02 23:09:32.035243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:09:32.035259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:11:41.461136] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:11:41.463241] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:11:41.465939] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:12:59.064087] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:12:59.064139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:12:59.064204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:09:33.441154 (xid=0x1864x) >[2012-10-02 23:12:59.064225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:12:59.064242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:16:26.094106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:16:26.094157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:16:26.094224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:13:00.471165 (xid=0x1865x) >[2012-10-02 23:16:26.094246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:16:26.094263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:19:53.121120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:19:53.121172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:19:53.121240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:16:27.498693 (xid=0x1866x) >[2012-10-02 23:19:53.121261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:19:53.121278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:21:41.544723] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:21:41.547596] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:21:41.549713] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:23:20.150146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:23:20.150207] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:23:20.150283] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:19:54.527860 (xid=0x1867x) >[2012-10-02 23:23:20.150305] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:23:20.150322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:26:47.181106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:26:47.181157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:26:47.181223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:23:21.558565 (xid=0x1868x) >[2012-10-02 23:26:47.181245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:26:47.181261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:30:14.210109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:30:14.210161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:30:14.210228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:26:48.588044 (xid=0x1869x) >[2012-10-02 23:30:14.210250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:30:14.210267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:31:41.632967] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:31:41.635464] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:31:41.637718] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:33:41.240130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:33:41.240182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:33:41.240265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:30:15.617837 (xid=0x1870x) >[2012-10-02 23:33:41.240286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:33:41.240302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:37:08.271107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:37:08.271161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:37:08.271237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:33:42.648985 (xid=0x1871x) >[2012-10-02 23:37:08.272200] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:37:08.272242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:40:35.301067] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:40:35.301121] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:40:35.301192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:37:09.678022 (xid=0x1872x) >[2012-10-02 23:40:35.301215] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:40:35.301231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:41:41.719070] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:41:41.722274] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:41:41.724788] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:44:02.330148] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:44:02.330195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:44:02.330260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:40:36.706870 (xid=0x1873x) >[2012-10-02 23:44:02.330291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:44:02.330308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:47:29.361096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:47:29.361149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:47:29.361217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:44:03.738043 (xid=0x1874x) >[2012-10-02 23:47:29.361238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:47:29.361254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:50:56.390111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:50:56.390160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:50:56.390228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:47:30.767438 (xid=0x1875x) >[2012-10-02 23:50:56.390249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:50:56.390265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:51:41.807150] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:51:41.810404] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:51:41.813025] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-02 23:54:23.420103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:54:23.420156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:54:23.420226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:50:57.796860 (xid=0x1876x) >[2012-10-02 23:54:23.420247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:54:23.420263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-02 23:57:50.450114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-02 23:57:50.450166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-02 23:57:50.450232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:54:24.827014 (xid=0x1877x) >[2012-10-02 23:57:50.450254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-02 23:57:50.450271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:01:17.479119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:01:17.479173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:01:17.479245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-02 23:57:51.855866 (xid=0x1878x) >[2012-10-03 00:01:17.479267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:01:17.479284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:01:41.891912] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:01:41.895281] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:01:41.898373] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:04:44.508110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:04:44.508163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:04:44.508235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:01:18.885013 (xid=0x1879x) >[2012-10-03 00:04:44.508256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:04:44.508272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:08:11.537120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:08:11.537171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:08:11.537239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:04:45.914768 (xid=0x1880x) >[2012-10-03 00:08:11.537273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:08:11.537292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:11:38.567140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:11:38.567194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:11:38.567262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:08:12.944104 (xid=0x1881x) >[2012-10-03 00:11:38.567284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:11:38.567300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:11:41.978293] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:11:41.981704] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:11:41.984692] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:15:05.596102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:15:05.596153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:15:05.596217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:11:39.973054 (xid=0x1882x) >[2012-10-03 00:15:05.596238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:15:05.596253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:18:32.629138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:18:32.629195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:18:32.629269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:15:07.006298 (xid=0x1883x) >[2012-10-03 00:18:32.629306] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:18:32.629324] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:21:42.062474] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:21:42.065772] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:21:42.068286] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:21:59.658079] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:21:59.658128] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:21:59.658202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:18:34.035346 (xid=0x1884x) >[2012-10-03 00:21:59.659165] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:21:59.659207] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:25:26.688110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:25:26.688170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:25:26.688244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:22:01.065412 (xid=0x1885x) >[2012-10-03 00:25:26.688266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:25:26.688283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:28:53.717075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:28:53.717127] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:28:53.717192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:25:28.094180 (xid=0x1886x) >[2012-10-03 00:28:53.717213] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:28:53.717228] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:31:42.153127] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:31:42.156251] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:31:42.158731] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:32:20.751116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:32:20.751170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:32:20.751242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:28:55.128025 (xid=0x1887x) >[2012-10-03 00:32:20.751263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:32:20.751280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:35:47.783120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:35:47.783172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:35:47.783241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:32:22.160067 (xid=0x1888x) >[2012-10-03 00:35:47.783263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:35:47.783279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:39:14.821101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:39:14.821151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:39:14.821216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:35:49.198892 (xid=0x1889x) >[2012-10-03 00:39:14.821237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:39:14.821253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:41:42.248237] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:41:42.251567] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:41:42.254139] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:42:41.850136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:42:41.850182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:42:41.850247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:39:16.227062 (xid=0x1890x) >[2012-10-03 00:42:41.850267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:42:41.850283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:46:08.880112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:46:08.880165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:46:08.880232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:42:43.257796 (xid=0x1891x) >[2012-10-03 00:46:08.880253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:46:08.880270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:49:35.909133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:49:35.909187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:49:35.909254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:46:10.285912 (xid=0x1892x) >[2012-10-03 00:49:35.909276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:49:35.909293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:51:42.334094] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:51:42.337373] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:51:42.339638] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 00:53:02.938123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:53:02.938171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:53:02.938234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:49:37.314967 (xid=0x1893x) >[2012-10-03 00:53:02.938255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:53:02.938271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:56:29.968134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:56:29.968186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:56:29.968255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:53:04.345124 (xid=0x1894x) >[2012-10-03 00:56:29.968277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:56:29.968295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 00:59:56.996095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 00:59:56.996151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 00:59:56.996225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:56:31.373215 (xid=0x1895x) >[2012-10-03 00:59:56.996877] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 00:59:56.996914] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:01:42.416036] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:01:42.419097] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:01:42.421938] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:03:24.023136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:03:24.023185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:03:24.023264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 00:59:58.400339 (xid=0x1896x) >[2012-10-03 01:03:24.023285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:03:24.023300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:06:51.053122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:06:51.053173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:06:51.053240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:03:25.429894 (xid=0x1897x) >[2012-10-03 01:06:51.053262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:06:51.053278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:10:18.082120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:10:18.082172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:10:18.082240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:06:52.460020 (xid=0x1898x) >[2012-10-03 01:10:18.082262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:10:18.082279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:11:42.500286] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:11:42.503557] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:11:42.506120] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:13:45.110112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:13:45.110161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:13:45.110227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:10:19.487076 (xid=0x1899x) >[2012-10-03 01:13:45.110260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:13:45.110277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:17:12.139137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:17:12.139187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:17:12.139252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:13:46.515897 (xid=0x1900x) >[2012-10-03 01:17:12.139273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:17:12.139288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:20:39.167138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:20:39.167190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:20:39.167256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:17:13.544382 (xid=0x1901x) >[2012-10-03 01:20:39.167277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:20:39.167293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:21:42.587229] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:21:42.590688] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:21:42.593190] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:24:06.197131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:24:06.197183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:24:06.197249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:20:40.573844 (xid=0x1902x) >[2012-10-03 01:24:06.197270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:24:06.197287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:27:33.229104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:27:33.229159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:27:33.229226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:24:07.606477 (xid=0x1903x) >[2012-10-03 01:27:33.229248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:27:33.229265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:31:00.261145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:31:00.261197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:31:00.261264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:27:34.638308 (xid=0x1904x) >[2012-10-03 01:31:00.261286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:31:00.261303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:31:42.676835] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:31:42.679761] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:31:42.682096] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:34:27.291116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:34:27.291178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:34:27.291252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:31:01.668031 (xid=0x1905x) >[2012-10-03 01:34:27.291274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:34:27.291291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:37:54.322118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:37:54.322170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:37:54.322252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:34:28.699285 (xid=0x1906x) >[2012-10-03 01:37:54.322273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:37:54.322289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:41:21.351101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:41:21.351155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:41:21.351219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:37:55.727821 (xid=0x1907x) >[2012-10-03 01:41:21.351241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:41:21.351257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:41:42.764175] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:41:42.766872] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:41:42.769068] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:44:48.380120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:44:48.380172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:44:48.380240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:41:22.757483 (xid=0x1908x) >[2012-10-03 01:44:48.380261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:44:48.380278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:48:15.412095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:48:15.412147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:48:15.412215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:44:49.789020 (xid=0x1909x) >[2012-10-03 01:48:15.412249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:48:15.412266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:51:42.441135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:51:42.441191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:51:42.441259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:48:16.818081 (xid=0x1910x) >[2012-10-03 01:51:42.441280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:51:42.441297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:51:42.847784] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:51:42.851059] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:51:42.854362] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 01:55:09.472103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:55:09.472152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:55:09.472220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:51:43.849225 (xid=0x1911x) >[2012-10-03 01:55:09.472241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:55:09.472258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 01:58:36.500138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 01:58:36.500191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 01:58:36.500259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:55:10.877638 (xid=0x1912x) >[2012-10-03 01:58:36.500281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 01:58:36.500298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:01:42.935422] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:01:42.939075] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:01:42.941469] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:02:03.530047] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:02:03.530092] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:02:03.530157] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 01:58:37.906865 (xid=0x1913x) >[2012-10-03 02:02:03.530178] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:02:03.530193] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:05:30.561130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:05:30.561189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:05:30.561265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:02:04.938074 (xid=0x1914x) >[2012-10-03 02:05:30.561287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:05:30.561305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:08:57.590124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:08:57.590178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:08:57.590245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:05:31.967542 (xid=0x1915x) >[2012-10-03 02:08:57.590266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:08:57.590283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:11:43.024131] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:11:43.026342] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:11:43.028904] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:12:24.621127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:12:24.621182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:12:24.621258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:08:58.997845 (xid=0x1916x) >[2012-10-03 02:12:24.621280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:12:24.621297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:15:51.654126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:15:51.654176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:15:51.654243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:12:26.032042 (xid=0x1917x) >[2012-10-03 02:15:51.654265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:15:51.654281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:19:18.682098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:19:18.682147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:19:18.682213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:15:53.059015 (xid=0x1918x) >[2012-10-03 02:19:18.682235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:19:18.682251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:21:43.114005] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:21:43.117551] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:21:43.119768] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:22:45.713104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:22:45.713156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:22:45.713223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:19:20.090030 (xid=0x1919x) >[2012-10-03 02:22:45.713245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:22:45.713261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:26:12.748125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:26:12.748176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:26:12.748245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:22:47.124886 (xid=0x1920x) >[2012-10-03 02:26:12.748266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:26:12.748282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:29:39.775141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:29:39.775193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:29:39.775261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:26:14.152705 (xid=0x1921x) >[2012-10-03 02:29:39.775282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:29:39.775299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:31:43.205457] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:31:43.208125] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:31:43.210256] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:33:06.808106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:33:06.808155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:33:06.808237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:29:41.185036 (xid=0x1922x) >[2012-10-03 02:33:06.808258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:33:06.808273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:36:33.842115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:36:33.842177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:36:33.842254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:33:08.219726 (xid=0x1923x) >[2012-10-03 02:36:33.842276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:36:33.842293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:40:00.875136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:40:00.875188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:40:00.875256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:36:35.251822 (xid=0x1924x) >[2012-10-03 02:40:00.875278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:40:00.875294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:41:43.301842] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:41:43.305511] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:41:43.308169] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:43:27.906122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:43:27.906177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:43:27.906255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:40:02.282923 (xid=0x1925x) >[2012-10-03 02:43:27.907071] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:43:27.907105] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:46:54.938110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:46:54.938161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:46:54.938229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:43:29.315649 (xid=0x1926x) >[2012-10-03 02:46:54.938250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:46:54.938267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:50:21.970135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:50:21.970183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:50:21.970249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:46:56.347521 (xid=0x1927x) >[2012-10-03 02:50:21.970269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:50:21.970285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:51:43.393239] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:51:43.395818] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:51:43.398160] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 02:53:49.001124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:53:49.001176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:53:49.001245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:50:23.378321 (xid=0x1928x) >[2012-10-03 02:53:49.001266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:53:49.001296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 02:57:16.033128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 02:57:16.033179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 02:57:16.033244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:53:50.410998 (xid=0x1929x) >[2012-10-03 02:57:16.033266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 02:57:16.033283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:00:43.064115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:00:43.064166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:00:43.064233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 02:57:17.441064 (xid=0x1930x) >[2012-10-03 03:00:43.064255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:00:43.064271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:01:43.485889] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:01:43.489710] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:01:43.492205] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:04:10.095071] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:04:10.095122] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:04:10.095190] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:00:44.472020 (xid=0x1931x) >[2012-10-03 03:04:10.095211] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:04:10.095227] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:07:37.126103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:07:37.126161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:07:37.126252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:04:11.503102 (xid=0x1932x) >[2012-10-03 03:07:37.126274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:07:37.126291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:11:04.155129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:11:04.155181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:11:04.155248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:07:38.532431 (xid=0x1933x) >[2012-10-03 03:11:04.155269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:11:04.155286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:11:43.573158] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:11:43.576371] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:11:43.579098] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:14:31.186130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:14:31.186191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:14:31.186270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:11:05.563070 (xid=0x1934x) >[2012-10-03 03:14:31.186293] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:14:31.186310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:17:58.219098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:17:58.219150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:17:58.219215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:14:32.596099 (xid=0x1935x) >[2012-10-03 03:17:58.219251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:17:58.219268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:21:25.248135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:21:25.248187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:21:25.248254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:17:59.625514 (xid=0x1936x) >[2012-10-03 03:21:25.248275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:21:25.248292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:21:43.661919] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:21:43.665104] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:21:43.667529] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:24:52.277125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:24:52.277177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:24:52.277246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:21:26.655007 (xid=0x1937x) >[2012-10-03 03:24:52.277268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:24:52.277285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:28:19.308098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:28:19.308154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:28:19.308226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:24:53.684831 (xid=0x1938x) >[2012-10-03 03:28:19.308247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:28:19.308264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:31:43.741643] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:31:43.744925] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:31:43.747671] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:31:46.335099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:31:46.335142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:31:46.335206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:28:20.712207 (xid=0x1939x) >[2012-10-03 03:31:46.335227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:31:46.335243] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:35:13.366100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:35:13.366147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:35:13.366212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:31:47.743680 (xid=0x1940x) >[2012-10-03 03:35:13.366233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:35:13.366248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:38:40.394110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:38:40.394165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:38:40.394234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:35:14.771069 (xid=0x1941x) >[2012-10-03 03:38:40.394255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:38:40.394272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:41:43.829798] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:41:43.833394] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:41:43.836222] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:42:07.424100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:42:07.424143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:42:07.424207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:38:41.802034 (xid=0x1942x) >[2012-10-03 03:42:07.424227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:42:07.424242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:45:34.456116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:45:34.456175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:45:34.456253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:42:08.833388 (xid=0x1943x) >[2012-10-03 03:45:34.456276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:45:34.456294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:49:01.486103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:49:01.486152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:49:01.486220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:45:35.863072 (xid=0x1944x) >[2012-10-03 03:49:01.486242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:49:01.486258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:51:43.919544] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:51:43.922963] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:51:43.925739] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 03:52:28.516109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:52:28.516161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:52:28.516228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:49:02.893026 (xid=0x1945x) >[2012-10-03 03:52:28.516249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:52:28.516265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:55:55.549127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:55:55.549178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:55:55.549244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:52:29.926667 (xid=0x1946x) >[2012-10-03 03:55:55.549265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:55:55.549281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 03:59:22.580102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 03:59:22.580155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 03:59:22.580220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:55:56.957321 (xid=0x1947x) >[2012-10-03 03:59:22.580242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 03:59:22.580258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:01:44.011727] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:01:44.014941] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:01:44.017599] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:02:49.611112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:02:49.611175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:02:49.611242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 03:59:23.988422 (xid=0x1948x) >[2012-10-03 04:02:49.611262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:02:49.611278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:06:16.646064] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:06:16.646117] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:06:16.646181] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:02:51.023542 (xid=0x1949x) >[2012-10-03 04:06:16.646203] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:06:16.646220] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:09:43.678111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:09:43.678163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:09:43.678232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:06:18.055739 (xid=0x1950x) >[2012-10-03 04:09:43.678253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:09:43.678269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:11:44.107677] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:11:44.110966] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:11:44.113596] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:13:10.710100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:13:10.710148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:13:10.710214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:09:45.087272 (xid=0x1951x) >[2012-10-03 04:13:10.710245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:13:10.710261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:16:37.744121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:16:37.744171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:16:37.744237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:13:12.121282 (xid=0x1952x) >[2012-10-03 04:16:37.744258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:16:37.744274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:20:04.775111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:20:04.775159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:20:04.775225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:16:39.151889 (xid=0x1953x) >[2012-10-03 04:20:04.775246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:20:04.775263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:21:44.203456] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:21:44.206841] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:21:44.209615] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:23:31.808122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:23:31.808172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:23:31.808239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:20:06.185051 (xid=0x1954x) >[2012-10-03 04:23:31.808260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:23:31.808288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:26:58.843125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:26:58.843177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:26:58.843245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:23:33.220195 (xid=0x1955x) >[2012-10-03 04:26:58.843268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:26:58.843285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:30:25.877112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:30:25.877164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:30:25.877233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:27:00.254316 (xid=0x1956x) >[2012-10-03 04:30:25.877254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:30:25.877271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:31:44.301951] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:31:44.305096] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:31:44.307524] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:33:52.910112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:33:52.910165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:33:52.910232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:30:27.287384 (xid=0x1957x) >[2012-10-03 04:33:52.910254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:33:52.910270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:37:19.947018] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:37:19.947080] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:37:19.947144] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:33:54.325043 (xid=0x1958x) >[2012-10-03 04:37:19.947165] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:37:19.947181] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:40:46.979138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:40:46.979195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:40:46.979261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:37:21.356252 (xid=0x1959x) >[2012-10-03 04:40:46.979283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:40:46.979299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:41:44.399467] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:41:44.402058] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:41:44.404213] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:44:14.009079] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:44:14.009133] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:44:14.009202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:40:48.386743 (xid=0x1960x) >[2012-10-03 04:44:14.010177] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:44:14.010221] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:47:41.042132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:47:41.042183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:47:41.042251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:44:15.419431 (xid=0x1961x) >[2012-10-03 04:47:41.042283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:47:41.042301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:51:08.072129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:51:08.072181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:51:08.072248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:47:42.449256 (xid=0x1962x) >[2012-10-03 04:51:08.072269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:51:08.072286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:51:44.488973] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:51:44.492010] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:51:44.494780] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 04:54:35.103115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:54:35.103168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:54:35.103236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:51:09.480276 (xid=0x1963x) >[2012-10-03 04:54:35.103257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:54:35.103274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 04:58:02.135121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 04:58:02.135171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 04:58:02.135237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:54:36.512643 (xid=0x1964x) >[2012-10-03 04:58:02.135259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 04:58:02.135286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:01:29.166125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:01:29.166186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:01:29.166270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 04:58:03.543426 (xid=0x1965x) >[2012-10-03 05:01:29.167282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:01:29.167322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:01:44.580764] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:01:44.583718] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:01:44.586458] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:04:56.198109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:04:56.198161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:04:56.198228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:01:30.574869 (xid=0x1966x) >[2012-10-03 05:04:56.198250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:04:56.198267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:08:23.231124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:08:23.231173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:08:23.231239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:04:57.608294 (xid=0x1967x) >[2012-10-03 05:08:23.231260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:08:23.231276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:11:44.662651] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:11:44.665301] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:11:44.667908] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:11:50.258097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:11:50.258140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:11:50.258202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:08:24.635604 (xid=0x1968x) >[2012-10-03 05:11:50.258222] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:11:50.258237] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:15:17.289122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:15:17.289172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:15:17.289237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:11:51.665889 (xid=0x1969x) >[2012-10-03 05:15:17.289258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:15:17.289274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:18:44.320125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:18:44.320177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:18:44.320246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:15:18.697482 (xid=0x1970x) >[2012-10-03 05:18:44.320267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:18:44.320284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:21:44.751173] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:21:44.754253] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:21:44.756484] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:22:11.348373] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:22:11.348418] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:22:11.348480] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:18:45.725348 (xid=0x1971x) >[2012-10-03 05:22:11.348500] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:22:11.348515] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:25:38.378124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:25:38.378179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:25:38.378245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:22:12.755081 (xid=0x1972x) >[2012-10-03 05:25:38.378267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:25:38.378283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:29:05.406123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:29:05.406173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:29:05.406239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:25:39.783693 (xid=0x1973x) >[2012-10-03 05:29:05.406260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:29:05.406276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:31:44.836417] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:31:44.839180] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:31:44.841437] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:32:32.435130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:32:32.435188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:32:32.435261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:29:06.812728 (xid=0x1974x) >[2012-10-03 05:32:32.435282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:32:32.435299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:35:59.469105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:35:59.469157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:35:59.469224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:32:33.846253 (xid=0x1975x) >[2012-10-03 05:35:59.469246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:35:59.469263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:39:26.497110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:39:26.497162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:39:26.497228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:36:00.874890 (xid=0x1976x) >[2012-10-03 05:39:26.497250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:39:26.497267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:41:44.927052] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:41:44.930472] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:41:44.933516] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:42:53.528117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:42:53.528166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:42:53.528234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:39:27.905080 (xid=0x1977x) >[2012-10-03 05:42:53.528270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:42:53.528287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:46:20.559137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:46:20.559189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:46:20.559256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:42:54.936412 (xid=0x1978x) >[2012-10-03 05:46:20.559278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:46:20.559294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:49:47.589090] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:49:47.589140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:49:47.589208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:46:21.966558 (xid=0x1979x) >[2012-10-03 05:49:47.589230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:49:47.589247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:51:45.015883] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:51:45.019061] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:51:45.021535] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 05:53:14.619128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:53:14.619178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:53:14.619243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:49:48.996777 (xid=0x1980x) >[2012-10-03 05:53:14.619277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:53:14.619293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 05:56:41.651144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 05:56:41.651197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 05:56:41.651266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:53:16.028462 (xid=0x1981x) >[2012-10-03 05:56:41.651288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 05:56:41.651305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:00:08.682137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:00:08.682187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:00:08.682254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 05:56:43.059638 (xid=0x1982x) >[2012-10-03 06:00:08.682276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:00:08.682292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:01:45.108598] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:01:45.111226] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:01:45.113386] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:03:35.714117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:03:35.714174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:03:35.714249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:00:10.091195 (xid=0x1983x) >[2012-10-03 06:03:35.714272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:03:35.714290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:07:02.747123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:07:02.747186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:07:02.747254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:03:37.124340 (xid=0x1984x) >[2012-10-03 06:07:02.747275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:07:02.747291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:10:29.779120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:10:29.779177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:10:29.779254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:07:04.156430 (xid=0x1985x) >[2012-10-03 06:10:29.780009] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:10:29.780046] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:11:45.203057] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:11:45.205835] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:11:45.207890] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:13:56.811124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:13:56.811173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:13:56.811239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:10:31.188797 (xid=0x1986x) >[2012-10-03 06:13:56.811260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:13:56.811276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:17:23.845105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:17:23.845155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:17:23.845221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:13:58.222075 (xid=0x1987x) >[2012-10-03 06:17:23.845253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:17:23.845270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:20:50.883123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:20:50.883174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:20:50.883241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:17:25.260760 (xid=0x1988x) >[2012-10-03 06:20:50.883262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:20:50.883279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:21:45.305352] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:21:45.308016] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:21:45.310208] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:24:17.916123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:24:17.916179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:24:17.916254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:20:52.293155 (xid=0x1989x) >[2012-10-03 06:24:17.917186] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:24:17.917232] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:27:44.951109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:27:44.951166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:27:44.951242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:24:19.328519 (xid=0x1990x) >[2012-10-03 06:27:44.951276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:27:44.951294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:31:11.983107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:31:11.983159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:31:11.983227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:27:46.360209 (xid=0x1991x) >[2012-10-03 06:31:11.983248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:31:11.983265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:31:45.396825] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:31:45.399802] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:31:45.401975] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:34:39.010020] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:34:39.010081] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:34:39.010156] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:31:13.387771 (xid=0x1992x) >[2012-10-03 06:34:39.010178] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:34:39.010195] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:38:06.040096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:38:06.040146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:38:06.040211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:34:40.417067 (xid=0x1993x) >[2012-10-03 06:38:06.040233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:38:06.040250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:41:33.069139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:41:33.069209] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:41:33.069284] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:38:07.446533 (xid=0x1994x) >[2012-10-03 06:41:33.069306] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:41:33.069322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:41:45.486355] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:41:45.489219] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:41:45.491701] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:45:00.103063] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:45:00.103117] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:45:00.103192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:41:34.480474 (xid=0x1995x) >[2012-10-03 06:45:00.122060] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:45:00.122104] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:48:27.136117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:48:27.136170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:48:27.136236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:45:01.513414 (xid=0x1996x) >[2012-10-03 06:48:27.136257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:48:27.136273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:51:45.577037] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:51:45.580017] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:51:45.582472] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 06:51:54.170100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:51:54.170141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:51:54.170205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:48:28.547329 (xid=0x1997x) >[2012-10-03 06:51:54.170225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:51:54.170241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:55:21.204142] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:55:21.204193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:55:21.204259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:51:55.580878 (xid=0x1998x) >[2012-10-03 06:55:21.204280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:55:21.204298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 06:58:48.235133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 06:58:48.235184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 06:58:48.235249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:55:22.612035 (xid=0x1999x) >[2012-10-03 06:58:48.235270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 06:58:48.235287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:01:45.671483] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:01:45.675104] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:01:45.677678] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:02:15.267109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:02:15.267153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:02:15.267217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 06:58:49.643898 (xid=0x2000x) >[2012-10-03 07:02:15.267237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:02:15.267252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:05:42.300134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:05:42.300185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:05:42.300254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:02:16.677148 (xid=0x2001x) >[2012-10-03 07:05:42.300275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:05:42.300291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:09:09.328054] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:09:09.328102] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:09:09.328176] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:05:43.705561 (xid=0x2002x) >[2012-10-03 07:09:09.328993] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:09:09.329029] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:11:45.760298] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:11:45.763414] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:11:45.765903] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:12:36.358125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:12:36.358172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:12:36.358250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:09:10.735736 (xid=0x2003x) >[2012-10-03 07:12:36.358271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:12:36.358286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:16:03.392111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:16:03.392162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:16:03.392228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:12:37.769903 (xid=0x2004x) >[2012-10-03 07:16:03.392250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:16:03.392266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:19:30.428137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:19:30.428188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:19:30.428254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:16:04.805566 (xid=0x2005x) >[2012-10-03 07:19:30.428277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:19:30.428293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:21:45.851705] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:21:45.854283] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:21:45.856569] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:22:57.455113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:22:57.455164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:22:57.455231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:19:31.832535 (xid=0x2006x) >[2012-10-03 07:22:57.455261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:22:57.455278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:26:24.484127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:26:24.484182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:26:24.484252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:22:58.861759 (xid=0x2007x) >[2012-10-03 07:26:24.484273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:26:24.484290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:29:51.513124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:29:51.513174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:29:51.513240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:26:25.890212 (xid=0x2008x) >[2012-10-03 07:29:51.513261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:29:51.513277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:31:45.938778] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:31:45.941672] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:31:45.943962] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:33:18.542126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:33:18.542172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:33:18.542237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:29:52.919601 (xid=0x2009x) >[2012-10-03 07:33:18.542258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:33:18.542274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:36:45.577113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:36:45.577171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:36:45.577245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:33:19.954155 (xid=0x2010x) >[2012-10-03 07:36:45.577268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:36:45.577285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:40:12.607124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:40:12.607173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:40:12.607242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:36:46.984608 (xid=0x2011x) >[2012-10-03 07:40:12.607264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:40:12.607280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:41:46.032619] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:41:46.035952] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:41:46.038810] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:43:39.639121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:43:39.639182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:43:39.639259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:40:14.015837 (xid=0x2012x) >[2012-10-03 07:43:39.639281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:43:39.639298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:47:06.672106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:47:06.672159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:47:06.672228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:43:41.049431 (xid=0x2013x) >[2012-10-03 07:47:06.672261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:47:06.672279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:50:33.703109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:50:33.703158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:50:33.703227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:47:08.080529 (xid=0x2014x) >[2012-10-03 07:50:33.703249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:50:33.703266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:51:46.128984] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:51:46.131965] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:51:46.134283] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 07:54:00.738108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:54:00.738158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:54:00.738224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:50:35.115428 (xid=0x2015x) >[2012-10-03 07:54:00.738245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:54:00.738261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 07:57:27.772127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 07:57:27.772177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 07:57:27.772246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:54:02.149291 (xid=0x2016x) >[2012-10-03 07:57:27.772281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 07:57:27.772299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:00:54.804106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:00:54.804158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:00:54.804225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 07:57:29.181327 (xid=0x2017x) >[2012-10-03 08:00:54.804247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:00:54.804263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:01:46.226470] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:01:46.229965] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:01:46.232837] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:04:21.837105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:04:21.837156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:04:21.837220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:00:56.214083 (xid=0x2018x) >[2012-10-03 08:04:21.837242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:04:21.837258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:07:48.873119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:07:48.873169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:07:48.873235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:04:23.250684 (xid=0x2019x) >[2012-10-03 08:07:48.873258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:07:48.873275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:11:15.907115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:11:15.907181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:11:15.907248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:07:50.284056 (xid=0x2020x) >[2012-10-03 08:11:15.907269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:11:15.907285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:11:46.326764] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:11:46.330064] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:11:46.332592] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:14:42.940134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:14:42.940195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:14:42.940271] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:11:17.317342 (xid=0x2021x) >[2012-10-03 08:14:42.940293] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:14:42.940311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:18:09.975112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:18:09.975164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:18:09.975231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:14:44.351968 (xid=0x2022x) >[2012-10-03 08:18:09.975253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:18:09.975270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:21:37.007120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:21:37.007181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:21:37.007256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:18:11.384129 (xid=0x2023x) >[2012-10-03 08:21:37.007295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:21:37.007313] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:21:46.422858] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:21:46.426321] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:21:46.428656] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:25:04.039145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:25:04.039194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:25:04.039262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:21:38.416740 (xid=0x2024x) >[2012-10-03 08:25:04.039283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:25:04.039300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:28:31.071119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:28:31.071174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:28:31.071249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:25:05.448508 (xid=0x2025x) >[2012-10-03 08:28:31.071999] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:28:31.072036] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:31:46.508332] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:31:46.511362] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:31:46.513827] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:31:58.101106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:31:58.101150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:31:58.101214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:28:32.478761 (xid=0x2026x) >[2012-10-03 08:31:58.101234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:31:58.101249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:35:25.134125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:35:25.134176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:35:25.134243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:31:59.511773 (xid=0x2027x) >[2012-10-03 08:35:25.134264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:35:25.134280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:38:52.164135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:38:52.164186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:38:52.164254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:35:26.541558 (xid=0x2028x) >[2012-10-03 08:38:52.164275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:38:52.164292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:41:46.599978] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:41:46.603545] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:41:46.606595] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:42:19.195131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:42:19.195180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:42:19.195261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:38:53.572249 (xid=0x2029x) >[2012-10-03 08:42:19.196245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:42:19.196296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:45:46.228122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:45:46.228186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:45:46.228263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:42:20.605314 (xid=0x2030x) >[2012-10-03 08:45:46.228285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:45:46.228303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:49:13.258118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:49:13.258165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:49:13.258230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:45:47.635999 (xid=0x2031x) >[2012-10-03 08:49:13.258251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:49:13.258267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:51:46.692536] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:51:46.696148] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:51:46.699074] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 08:52:40.292120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:52:40.292181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:52:40.292256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:49:14.669036 (xid=0x2032x) >[2012-10-03 08:52:40.292290] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:52:40.292308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:56:07.325142] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:56:07.325193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:56:07.325261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:52:41.702636 (xid=0x2033x) >[2012-10-03 08:56:07.325282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:56:07.325298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 08:59:34.353098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 08:59:34.353148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 08:59:34.353216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:56:08.730666 (xid=0x2034x) >[2012-10-03 08:59:34.353237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 08:59:34.353254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:01:47.073163] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:01:47.075986] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:01:47.078356] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:03:01.675105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:03:01.675154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:03:01.675221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 08:59:36.052778 (xid=0x2035x) >[2012-10-03 09:03:01.675243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:03:01.675259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:06:28.706117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:06:28.706169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:06:28.706234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:03:03.083052 (xid=0x2036x) >[2012-10-03 09:06:28.706255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:06:28.706271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:09:55.734117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:09:55.734171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:09:55.734243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:06:30.111532 (xid=0x2037x) >[2012-10-03 09:09:55.735004] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:09:55.735041] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:11:47.158598] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:11:47.162165] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:11:47.165559] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:13:22.763134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:13:22.763186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:13:22.763253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:09:57.139878 (xid=0x2038x) >[2012-10-03 09:13:22.763275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:13:22.763291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:16:49.794099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:16:49.794150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:16:49.794235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:13:24.171169 (xid=0x2039x) >[2012-10-03 09:16:49.794257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:16:49.794272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:20:16.823106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:20:16.823159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:20:16.823225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:16:51.200521 (xid=0x2040x) >[2012-10-03 09:20:16.823247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:20:16.823264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:21:47.246072] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:21:47.249280] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:21:47.251860] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:23:43.853124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:23:43.853176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:23:43.853245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:20:18.230031 (xid=0x2041x) >[2012-10-03 09:23:43.853267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:23:43.853284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:27:10.884107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:27:10.884157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:27:10.884224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:23:45.261615 (xid=0x2042x) >[2012-10-03 09:27:10.884256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:27:10.884274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:30:37.915105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:30:37.915168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:30:37.915248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:27:12.292313 (xid=0x2043x) >[2012-10-03 09:30:37.915270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:30:37.915288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:31:47.336665] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:31:47.339940] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:31:47.342967] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:34:04.946144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:34:04.946196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:34:04.946261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:30:39.323052 (xid=0x2044x) >[2012-10-03 09:34:04.946283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:34:04.946299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:37:31.977068] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:37:31.977120] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:37:31.977186] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:34:06.354417 (xid=0x2045x) >[2012-10-03 09:37:31.977207] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:37:31.977223] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:40:59.008137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:40:59.008187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:40:59.008255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:37:33.384892 (xid=0x2046x) >[2012-10-03 09:40:59.008276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:40:59.008293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:41:47.425513] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:41:47.428663] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:41:47.431391] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:44:26.037078] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:44:26.037129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:44:26.037196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:41:00.414607 (xid=0x2047x) >[2012-10-03 09:44:26.037218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:44:26.037234] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:47:53.068077] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:47:53.068129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:47:53.068198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:44:27.445696 (xid=0x2048x) >[2012-10-03 09:47:53.068220] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:47:53.068236] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:51:20.100130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:51:20.100180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:51:20.100263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:47:54.476875 (xid=0x2049x) >[2012-10-03 09:51:20.100284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:51:20.100299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:51:47.514856] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:51:47.518169] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:51:47.521189] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 09:54:47.130137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:54:47.130191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:54:47.130260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:51:21.507114 (xid=0x2050x) >[2012-10-03 09:54:47.130281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:54:47.130298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 09:58:14.160107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 09:58:14.160162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 09:58:14.160231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:54:48.537636 (xid=0x2051x) >[2012-10-03 09:58:14.160253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 09:58:14.160270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:01:41.191103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:01:41.191152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:01:41.191219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 09:58:15.567831 (xid=0x2052x) >[2012-10-03 10:01:41.191252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:01:41.191269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:01:47.603995] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:01:47.607164] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:01:47.610072] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:05:08.221136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:05:08.221190] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:05:08.221259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:01:42.598448 (xid=0x2053x) >[2012-10-03 10:05:08.221280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:05:08.221297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:08:35.253146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:08:35.253199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:08:35.253266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:05:09.630433 (xid=0x2054x) >[2012-10-03 10:08:35.253287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:08:35.253304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:11:47.688797] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:11:47.692361] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:11:47.695482] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:12:02.284128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:12:02.284174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:12:02.284248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:08:36.661516 (xid=0x2055x) >[2012-10-03 10:12:02.284268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:12:02.284283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:15:29.316144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:15:29.316194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:15:29.316261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:12:03.692989 (xid=0x2056x) >[2012-10-03 10:15:29.316282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:15:29.316299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:18:56.345104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:18:56.345155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:18:56.345222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:15:30.722107 (xid=0x2057x) >[2012-10-03 10:18:56.345243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:18:56.345260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:21:47.776533] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:21:47.780021] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:21:47.782694] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:22:23.373095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:22:23.373141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:22:23.373204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:18:57.750789 (xid=0x2058x) >[2012-10-03 10:22:23.373237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:22:23.373254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:25:50.407144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:25:50.407203] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:25:50.407278] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:22:24.784504 (xid=0x2059x) >[2012-10-03 10:25:50.407300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:25:50.407318] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:29:17.437122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:29:17.437172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:29:17.437239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:25:51.814071 (xid=0x2060x) >[2012-10-03 10:29:17.437261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:29:17.437278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:31:47.867349] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:31:47.870268] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:31:47.872430] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:32:44.466133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:32:44.466179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:32:44.466243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:29:18.843368 (xid=0x2061x) >[2012-10-03 10:32:44.466264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:32:44.466291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:36:11.501109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:36:11.501159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:36:11.501224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:32:45.878380 (xid=0x2062x) >[2012-10-03 10:36:11.501245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:36:11.501261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:39:38.537133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:39:38.537182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:39:38.537250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:36:12.913894 (xid=0x2063x) >[2012-10-03 10:39:38.537271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:39:38.537289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:41:47.966099] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:41:47.969605] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:41:47.972409] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:43:05.567060] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:43:05.567113] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:43:05.567180] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:39:39.944635 (xid=0x2064x) >[2012-10-03 10:43:05.567201] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:43:05.567217] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:46:32.602154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:46:32.602199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:46:32.602274] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:43:06.979202 (xid=0x2065x) >[2012-10-03 10:46:32.602295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:46:32.602311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:49:59.632124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:49:59.632175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:49:59.632242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:46:34.009049 (xid=0x2066x) >[2012-10-03 10:49:59.632264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:49:59.632280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:51:48.058452] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:51:48.061755] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:51:48.064675] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 10:53:26.662111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:53:26.662170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:53:26.662245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:50:01.039788 (xid=0x2067x) >[2012-10-03 10:53:26.662266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:53:26.662283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 10:56:53.698088] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 10:56:53.698140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 10:56:53.698205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:53:28.074816 (xid=0x2068x) >[2012-10-03 10:56:53.698237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 10:56:53.698254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:00:20.729122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:00:20.729171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:00:20.729238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 10:56:55.106021 (xid=0x2069x) >[2012-10-03 11:00:20.729259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:00:20.729276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:01:48.153774] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:01:48.156871] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:01:48.159342] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:03:47.760116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:03:47.760173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:03:47.760244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:00:22.137132 (xid=0x2070x) >[2012-10-03 11:03:47.760266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:03:47.760283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:07:14.794103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:07:14.794153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:07:14.794219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:03:49.171660 (xid=0x2071x) >[2012-10-03 11:07:14.794241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:07:14.794269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:10:41.826123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:10:41.826177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:10:41.826248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:07:16.203228 (xid=0x2072x) >[2012-10-03 11:10:41.826270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:10:41.826287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:11:48.248085] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:11:48.251412] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:11:48.253843] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:14:08.857134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:14:08.857183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:14:08.857249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:10:43.234571 (xid=0x2073x) >[2012-10-03 11:14:08.857271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:14:08.857287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:17:35.890131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:17:35.890183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:17:35.890249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:14:10.267789 (xid=0x2074x) >[2012-10-03 11:17:35.890270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:17:35.890286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:21:02.920140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:21:02.920188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:21:02.920268] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:17:37.297533 (xid=0x2075x) >[2012-10-03 11:21:02.920289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:21:02.920305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:21:48.336816] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:21:48.339888] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:21:48.342367] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:24:29.949127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:24:29.949176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:24:29.949243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:21:04.326284 (xid=0x2076x) >[2012-10-03 11:24:29.949264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:24:29.949280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:27:56.981118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:27:56.981166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:27:56.981232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:24:31.358028 (xid=0x2077x) >[2012-10-03 11:27:56.981254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:27:56.981271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:31:24.010131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:31:24.010181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:31:24.010247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:27:58.387317 (xid=0x2078x) >[2012-10-03 11:31:24.010279] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:31:24.010297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:31:48.424509] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:31:48.427369] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:31:48.429888] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:34:51.040034] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:34:51.040094] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:34:51.040159] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:31:25.416855 (xid=0x2079x) >[2012-10-03 11:34:51.040180] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:34:51.040204] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:38:18.072131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:38:18.072181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:38:18.072247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:34:52.448921 (xid=0x2080x) >[2012-10-03 11:38:18.072268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:38:18.072284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:41:45.101141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:41:45.101200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:41:45.101273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:38:19.478614 (xid=0x2081x) >[2012-10-03 11:41:45.101295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:41:45.101313] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:41:48.513478] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:41:48.516210] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:41:48.518324] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:45:12.131117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:45:12.131166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:45:12.131234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:41:46.508653 (xid=0x2082x) >[2012-10-03 11:45:12.131255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:45:12.131272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:48:39.165138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:48:39.165191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:48:39.165258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:45:13.542736 (xid=0x2083x) >[2012-10-03 11:48:39.165280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:48:39.165296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:51:48.602077] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:51:48.604895] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:51:48.607010] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 11:52:06.196104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:52:06.196149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:52:06.196211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:48:40.573138 (xid=0x2084x) >[2012-10-03 11:52:06.196242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:52:06.196258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:55:33.228119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:55:33.228168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:55:33.228235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:52:07.605409 (xid=0x2085x) >[2012-10-03 11:55:33.228256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:55:33.228273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 11:59:00.551111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 11:59:00.551160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 11:59:00.551226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:55:34.927861 (xid=0x2086x) >[2012-10-03 11:59:00.551247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 11:59:00.551264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:01:48.984678] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:01:48.987996] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:01:48.990756] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:02:27.581109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:02:27.581154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:02:27.581219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 11:59:01.958397 (xid=0x2087x) >[2012-10-03 12:02:27.581239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:02:27.581267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:05:54.614119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:05:54.614177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:05:54.614250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:02:28.991208 (xid=0x2088x) >[2012-10-03 12:05:54.614272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:05:54.614289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:09:21.646098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:09:21.646147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:09:21.646215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:05:56.023375 (xid=0x2089x) >[2012-10-03 12:09:21.646236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:09:21.646253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:11:49.076904] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:11:49.080137] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:11:49.082816] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:12:48.676110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:12:48.676155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:12:48.676218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:09:23.053638 (xid=0x2090x) >[2012-10-03 12:12:48.676239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:12:48.676255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:16:15.707111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:16:15.707173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:16:15.707238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:12:50.084781 (xid=0x2091x) >[2012-10-03 12:16:15.707259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:16:15.707275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:19:42.742105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:19:42.742153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:19:42.742218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:16:17.119602 (xid=0x2092x) >[2012-10-03 12:19:42.742238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:19:42.742254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:21:49.170203] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:21:49.172817] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:21:49.175101] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:23:09.773105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:23:09.773157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:23:09.773223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:19:44.150048 (xid=0x2093x) >[2012-10-03 12:23:09.773245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:23:09.773262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:26:36.805126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:26:36.805175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:26:36.805250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:23:11.182049 (xid=0x2094x) >[2012-10-03 12:26:36.806033] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:26:36.806068] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:30:03.838136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:30:03.838186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:30:03.838250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:26:38.215180 (xid=0x2095x) >[2012-10-03 12:30:03.838272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:30:03.838288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:31:49.262892] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:31:49.265704] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:31:49.268104] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:33:30.868130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:33:30.868178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:33:30.868246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:30:05.245409 (xid=0x2096x) >[2012-10-03 12:33:30.868267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:33:30.868284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:36:57.902123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:36:57.902179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:36:57.902256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:33:32.279780 (xid=0x2097x) >[2012-10-03 12:36:57.902278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:36:57.902307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:40:24.934123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:40:24.934173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:40:24.934240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:36:59.311433 (xid=0x2098x) >[2012-10-03 12:40:24.934261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:40:24.934278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:41:49.356200] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:41:49.359639] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:41:49.362529] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:43:51.964120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:43:51.964174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:43:51.964256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:40:26.341284 (xid=0x2099x) >[2012-10-03 12:43:51.965071] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:43:51.965114] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:47:18.996131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:47:18.996184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:47:18.996256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:43:53.372999 (xid=0x2100x) >[2012-10-03 12:47:18.996277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:47:18.996294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:50:46.026129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:50:46.026188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:50:46.026276] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:47:20.403059 (xid=0x2101x) >[2012-10-03 12:50:46.026298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:50:46.026316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:51:49.443834] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:51:49.447173] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:51:49.449724] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 12:54:13.054128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:54:13.054179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:54:13.054247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:50:47.431178 (xid=0x2102x) >[2012-10-03 12:54:13.054269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:54:13.054286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 12:57:40.086137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 12:57:40.086186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 12:57:40.086251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:54:14.463277 (xid=0x2103x) >[2012-10-03 12:57:40.086272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 12:57:40.086287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:01:07.409105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:01:07.409154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:01:07.409222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 12:57:41.786367 (xid=0x2104x) >[2012-10-03 13:01:07.409257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:01:07.409274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:01:49.825628] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:01:49.828853] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:01:49.831233] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:04:34.438118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:04:34.438171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:04:34.438238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:01:08.815397 (xid=0x2105x) >[2012-10-03 13:04:34.438260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:04:34.438277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:08:01.471129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:08:01.471177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:08:01.471242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:04:35.848064 (xid=0x2106x) >[2012-10-03 13:08:01.471264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:08:01.471280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:11:28.502087] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:11:28.502137] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:11:28.502203] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:08:02.879008 (xid=0x2107x) >[2012-10-03 13:11:28.502224] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:11:28.502252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:11:49.915707] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:11:49.918370] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:11:49.920684] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:14:55.531108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:14:55.531161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:14:55.531229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:11:29.908736 (xid=0x2108x) >[2012-10-03 13:14:55.531251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:14:55.531269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:18:22.564105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:18:22.564155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:18:22.564221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:14:56.941366 (xid=0x2109x) >[2012-10-03 13:18:22.564242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:18:22.564259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:21:49.596104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:21:49.596163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:21:49.596237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:18:23.973126 (xid=0x2110x) >[2012-10-03 13:21:49.596260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:21:49.596277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:21:50.003692] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:21:50.007080] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:21:50.009770] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:25:16.629111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:25:16.629162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:25:16.629230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:21:51.005819 (xid=0x2111x) >[2012-10-03 13:25:16.629252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:25:16.629268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:28:43.659096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:28:43.659147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:28:43.659214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:25:18.036071 (xid=0x2112x) >[2012-10-03 13:28:43.659236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:28:43.659253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:31:50.096981] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:31:50.100095] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:31:50.102440] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:32:10.691103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:32:10.691145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:32:10.691209] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:28:45.067858 (xid=0x2113x) >[2012-10-03 13:32:10.691241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:32:10.691258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:35:37.724114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:35:37.724166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:35:37.724233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:32:12.101191 (xid=0x2114x) >[2012-10-03 13:35:37.724255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:35:37.724271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:39:04.753102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:39:04.753153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:39:04.753219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:35:39.130349 (xid=0x2115x) >[2012-10-03 13:39:04.753240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:39:04.753256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:41:50.185325] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:41:50.188484] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:41:50.191137] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:42:31.784071] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:42:31.784118] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:42:31.784180] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:39:06.161102 (xid=0x2116x) >[2012-10-03 13:42:31.784201] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:42:31.784217] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:45:58.815123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:45:58.815183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:45:58.815248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:42:33.192061 (xid=0x2117x) >[2012-10-03 13:45:58.815268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:45:58.815284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:49:25.846096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:49:25.846146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:49:25.846213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:46:00.223781 (xid=0x2118x) >[2012-10-03 13:49:25.846233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:49:25.846249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:51:50.570318] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:51:50.572948] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:51:50.575536] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 13:52:53.171099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:52:53.171150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:52:53.171221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:49:27.548037 (xid=0x2119x) >[2012-10-03 13:52:53.171243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:52:53.171260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:56:20.204109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:56:20.204157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:56:20.204223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:52:54.581694 (xid=0x2120x) >[2012-10-03 13:56:20.204256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:56:20.204274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 13:59:47.234121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 13:59:47.234178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 13:59:47.234251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:56:21.611663 (xid=0x2121x) >[2012-10-03 13:59:47.234274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 13:59:47.234291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:01:50.664135] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:01:50.667604] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:01:50.670385] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:03:14.266066] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:03:14.266117] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:03:14.266183] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 13:59:48.643602 (xid=0x2122x) >[2012-10-03 14:03:14.266205] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:03:14.266222] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:06:41.298051] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:06:41.298099] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:06:41.298163] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:03:15.675350 (xid=0x2123x) >[2012-10-03 14:06:41.298196] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:06:41.298214] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:10:08.327121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:10:08.327169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:10:08.327237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:06:42.704322 (xid=0x2124x) >[2012-10-03 14:10:08.327259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:10:08.327276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:11:50.752157] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:11:50.755157] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:11:50.757473] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:13:35.358113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:13:35.358163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:13:35.358230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:10:09.735046 (xid=0x2125x) >[2012-10-03 14:13:35.358252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:13:35.358268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:17:02.389089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:17:02.389148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:17:02.389220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:13:36.766050 (xid=0x2126x) >[2012-10-03 14:17:02.389242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:17:02.389259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:20:29.418104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:20:29.418162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:20:29.418230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:17:03.795607 (xid=0x2127x) >[2012-10-03 14:20:29.418250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:20:29.418267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:21:50.842078] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:21:50.845318] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:21:50.847659] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:23:56.449125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:23:56.449174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:23:56.449237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:20:30.826781 (xid=0x2128x) >[2012-10-03 14:23:56.449258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:23:56.449273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:27:23.481126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:27:23.481176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:27:23.481240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:23:57.858518 (xid=0x2129x) >[2012-10-03 14:27:23.481261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:27:23.481277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:30:50.511152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:30:50.511201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:30:50.511267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:27:24.888763 (xid=0x2130x) >[2012-10-03 14:30:50.511299] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:30:50.511317] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:31:51.224866] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:31:51.227984] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:31:51.230617] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:34:17.836133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:34:17.836186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:34:17.836253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:30:52.213526 (xid=0x2131x) >[2012-10-03 14:34:17.836275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:34:17.836292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:37:44.867144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:37:44.867192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:37:44.867257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:34:19.243973 (xid=0x2132x) >[2012-10-03 14:37:44.867278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:37:44.867294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:41:11.894131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:41:11.894181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:41:11.894247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:37:46.271761 (xid=0x2133x) >[2012-10-03 14:41:11.894269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:41:11.894296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:41:51.310515] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:41:51.314202] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:41:51.317361] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:44:38.924123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:44:38.924171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:44:38.924243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:41:13.301325 (xid=0x2134x) >[2012-10-03 14:44:38.925066] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:44:38.925109] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:48:05.954123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:48:05.954174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:48:05.954242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:44:40.330893 (xid=0x2135x) >[2012-10-03 14:48:05.954263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:48:05.954280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:51:32.981127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:51:32.981176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:51:32.981245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:48:07.358395 (xid=0x2136x) >[2012-10-03 14:51:32.981267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:51:32.981284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:51:51.686766] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:51:51.689742] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:51:51.692259] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 14:55:00.303151] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:55:00.303211] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:55:00.303286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:51:34.680052 (xid=0x2137x) >[2012-10-03 14:55:00.303308] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:55:00.303325] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 14:58:27.332097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 14:58:27.332147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 14:58:27.332214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:55:01.709353 (xid=0x2138x) >[2012-10-03 14:58:27.332235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 14:58:27.332252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:01:51.766858] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:01:51.770346] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:01:51.772934] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:01:54.361107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:01:54.361145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:01:54.361208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 14:58:28.737852 (xid=0x2139x) >[2012-10-03 15:01:54.361242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:01:54.361259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:05:21.392104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:05:21.392154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:05:21.392221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:01:55.769776 (xid=0x2140x) >[2012-10-03 15:05:21.392242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:05:21.392259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:08:48.421143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:08:48.421193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:08:48.421260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:05:22.798785 (xid=0x2141x) >[2012-10-03 15:08:48.421281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:08:48.421298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:11:51.856155] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:11:51.859731] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:11:51.862249] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:12:15.451101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:12:15.451148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:12:15.451211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:08:49.828303 (xid=0x2142x) >[2012-10-03 15:12:15.451232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:12:15.451248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:15:42.777108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:15:42.777159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:15:42.777226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:12:17.154769 (xid=0x2143x) >[2012-10-03 15:15:42.777248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:15:42.777265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:19:09.807105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:19:09.807155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:19:09.807220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:15:44.184759 (xid=0x2144x) >[2012-10-03 15:19:09.807242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:19:09.807257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:21:52.240999] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:21:52.244357] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:21:52.247497] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:22:36.837111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:22:36.837159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:22:36.837224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:19:11.214732 (xid=0x2145x) >[2012-10-03 15:22:36.837244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:22:36.837260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:26:03.873140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:26:03.873188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:26:03.873268] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:22:38.249833 (xid=0x2146x) >[2012-10-03 15:26:03.873289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:26:03.873306] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:29:30.903095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:29:30.903146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:29:30.903212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:26:05.280185 (xid=0x2147x) >[2012-10-03 15:29:30.903233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:29:30.903250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:31:52.333151] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:31:52.335939] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:31:52.337926] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:32:57.933103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:32:57.933158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:32:57.933231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:29:32.310397 (xid=0x2148x) >[2012-10-03 15:32:57.933253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:32:57.933270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:36:24.967122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:36:24.967174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:36:24.967239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:32:59.344382 (xid=0x2149x) >[2012-10-03 15:36:24.967271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:36:24.967288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:39:51.997090] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:39:51.997138] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:39:51.997204] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:36:26.374029 (xid=0x2150x) >[2012-10-03 15:39:51.997225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:39:51.997241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:41:52.421787] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:41:52.424733] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:41:52.427016] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:43:19.026130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:43:19.026182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:43:19.026247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:39:53.403068 (xid=0x2151x) >[2012-10-03 15:43:19.026268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:43:19.026284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:46:46.581102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:46:46.581154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:46:46.581220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:43:20.958268 (xid=0x2152x) >[2012-10-03 15:46:46.581241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:46:46.581257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:50:13.610145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:50:13.610209] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:50:13.610277] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:46:47.987373 (xid=0x2153x) >[2012-10-03 15:50:13.610298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:50:13.610314] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:51:53.033262] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:51:53.036719] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:51:53.039871] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 15:53:40.639141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:53:40.639191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:53:40.639255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:50:15.016492 (xid=0x2154x) >[2012-10-03 15:53:40.639276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:53:40.639292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 15:57:07.673124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 15:57:07.673180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 15:57:07.673254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:53:42.050199 (xid=0x2155x) >[2012-10-03 15:57:07.673276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 15:57:07.673292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:00:34.702102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:00:34.702154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:00:34.702222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 15:57:09.079752 (xid=0x2156x) >[2012-10-03 16:00:34.702254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:00:34.702271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:01:53.123965] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:01:53.126679] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:01:53.128858] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:04:01.732115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:04:01.732164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:04:01.732230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:00:36.109259 (xid=0x2157x) >[2012-10-03 16:04:01.732251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:04:01.732267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:07:29.058098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:07:29.058148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:07:29.058213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:04:03.435468 (xid=0x2158x) >[2012-10-03 16:07:29.058234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:07:29.058250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:10:56.086106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:10:56.086165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:10:56.086247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:07:30.463589 (xid=0x2159x) >[2012-10-03 16:10:56.087224] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:10:56.087261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:11:53.503150] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:11:53.505783] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:11:53.507842] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:14:23.114090] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:14:23.114141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:14:23.114206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:10:57.491270 (xid=0x2160x) >[2012-10-03 16:14:23.114228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:14:23.114246] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:17:50.148099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:17:50.148149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:17:50.148215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:14:24.525347 (xid=0x2161x) >[2012-10-03 16:17:50.148237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:17:50.148254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:21:17.176148] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:21:17.176201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:21:17.176270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:17:51.553189 (xid=0x2162x) >[2012-10-03 16:21:17.176291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:21:17.176308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:21:53.590047] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:21:53.593461] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:21:53.596262] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:24:44.204158] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:24:44.204210] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:24:44.204279] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:21:18.581301 (xid=0x2163x) >[2012-10-03 16:24:44.204300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:24:44.204317] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:28:11.236094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:28:11.236145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:28:11.236211] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:24:45.613557 (xid=0x2164x) >[2012-10-03 16:28:11.236233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:28:11.236249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:31:38.264163] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:31:38.264218] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:31:38.264290] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:28:12.641095 (xid=0x2165x) >[2012-10-03 16:31:38.264312] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:31:38.264328] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:31:53.676552] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:31:53.679039] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:31:53.681251] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:35:05.292173] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:35:05.292227] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:35:05.292300] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:31:39.669044 (xid=0x2166x) >[2012-10-03 16:35:05.292322] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:35:05.292339] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:38:32.617144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:38:32.617196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:38:32.617263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:35:06.994062 (xid=0x2167x) >[2012-10-03 16:38:32.617285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:38:32.617301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:41:54.056954] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:41:54.059769] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:41:54.062087] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:41:59.648126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:41:59.648168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:41:59.648240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:38:34.025033 (xid=0x2168x) >[2012-10-03 16:41:59.648261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:41:59.648277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:45:26.681069] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:45:26.681117] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:45:26.681183] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:42:01.058590 (xid=0x2169x) >[2012-10-03 16:45:26.681204] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:45:26.681220] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:48:54.011083] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:48:54.011142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:48:54.011216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:45:28.388607 (xid=0x2170x) >[2012-10-03 16:48:54.011238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:48:54.011255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:51:54.447677] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:51:54.450253] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:51:54.453147] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 16:52:21.042120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:52:21.042168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:52:21.042236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:48:55.419383 (xid=0x2171x) >[2012-10-03 16:52:21.042257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:52:21.042273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:55:48.077150] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:55:48.077201] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:55:48.077280] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:52:22.454120 (xid=0x2172x) >[2012-10-03 16:55:48.077301] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:55:48.077316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 16:59:15.110143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 16:59:15.110192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 16:59:15.110259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:55:49.487474 (xid=0x2173x) >[2012-10-03 16:59:15.110280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 16:59:15.110297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:01:54.541082] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:01:54.544579] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:01:54.547102] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:02:42.139126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:02:42.139179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:02:42.139247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 16:59:16.516715 (xid=0x2174x) >[2012-10-03 17:02:42.139268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:02:42.139284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:06:09.171140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:06:09.171197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:06:09.171270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:02:43.548519 (xid=0x2175x) >[2012-10-03 17:06:09.171304] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:06:09.171323] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:09:36.203107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:09:36.203160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:09:36.203228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:06:10.580339 (xid=0x2176x) >[2012-10-03 17:09:36.203249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:09:36.203266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:11:54.632180] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:11:54.635055] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:11:54.637725] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:13:03.233140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:13:03.233191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:13:03.233266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:09:37.610402 (xid=0x2177x) >[2012-10-03 17:13:03.233288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:13:03.233305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:16:30.265113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:16:30.265164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:16:30.265232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:13:04.642245 (xid=0x2178x) >[2012-10-03 17:16:30.265253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:16:30.265270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:19:57.297143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:19:57.297194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:19:57.297261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:16:31.674614 (xid=0x2179x) >[2012-10-03 17:19:57.297282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:19:57.297299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:21:54.722206] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:21:54.724944] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:21:54.727277] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:23:24.326111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:23:24.326159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:23:24.326224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:19:58.703150 (xid=0x2180x) >[2012-10-03 17:23:24.326245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:23:24.326260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:26:51.356091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:26:51.356139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:26:51.356203] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:23:25.733556 (xid=0x2181x) >[2012-10-03 17:26:51.356224] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:26:51.356241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:30:18.386096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:30:18.386148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:30:18.386233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:26:52.763657 (xid=0x2182x) >[2012-10-03 17:30:18.386254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:30:18.386270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:31:54.807844] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:31:54.811470] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:31:54.813960] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:33:45.413078] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:33:45.413125] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:33:45.413191] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:30:19.790715 (xid=0x2183x) >[2012-10-03 17:33:45.413211] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:33:45.413229] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:37:12.445136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:37:12.445187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:37:12.445255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:33:46.822264 (xid=0x2184x) >[2012-10-03 17:37:12.445277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:37:12.445293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:40:39.476111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:40:39.476160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:40:39.476227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:37:13.853387 (xid=0x2185x) >[2012-10-03 17:40:39.476260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:40:39.476277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:41:54.897758] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:41:54.901106] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:41:54.903954] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:44:06.507103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:44:06.507159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:44:06.507232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:40:40.884347 (xid=0x2186x) >[2012-10-03 17:44:06.507254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:44:06.507271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:47:33.545117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:47:33.545167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:47:33.545233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:44:07.922458 (xid=0x2187x) >[2012-10-03 17:47:33.545254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:47:33.545270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:51:00.574147] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:51:00.574200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:51:00.574276] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:47:34.951292 (xid=0x2188x) >[2012-10-03 17:51:00.574298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:51:00.574316] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:51:54.992259] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:51:54.995947] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:51:54.998377] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 17:54:27.603074] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:54:27.603129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:54:27.603196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:51:01.981031 (xid=0x2189x) >[2012-10-03 17:54:27.603217] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:54:27.603234] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 17:57:54.638152] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 17:57:54.638198] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 17:57:54.638265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:54:29.015258 (xid=0x2190x) >[2012-10-03 17:57:54.638287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 17:57:54.638303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:01:21.669105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:01:21.669155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:01:21.669222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 17:57:56.046530 (xid=0x2191x) >[2012-10-03 18:01:21.669244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:01:21.669261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:01:55.084601] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:01:55.088312] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:01:55.091360] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:04:48.698105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:04:48.698159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:04:48.698228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:01:23.075285 (xid=0x2192x) >[2012-10-03 18:04:48.698250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:04:48.698267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:08:15.734109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:08:15.734161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:08:15.734228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:04:50.112045 (xid=0x2193x) >[2012-10-03 18:08:15.734250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:08:15.734267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:11:42.767140] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:11:42.767191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:11:42.767258] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:08:17.143856 (xid=0x2194x) >[2012-10-03 18:11:42.767280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:11:42.767297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:11:55.179462] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:11:55.183056] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:11:55.185854] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:15:09.796115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:15:09.796164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:15:09.796231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:11:44.173102 (xid=0x2195x) >[2012-10-03 18:15:09.796252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:15:09.796268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:18:36.832116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:18:36.832166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:18:36.832232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:15:11.208840 (xid=0x2196x) >[2012-10-03 18:18:36.832253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:18:36.832270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:21:55.565147] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:21:55.568288] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:21:55.570956] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:22:04.159145] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:22:04.159191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:22:04.159255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:18:38.536490 (xid=0x2197x) >[2012-10-03 18:22:04.159276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:22:04.159291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:25:31.191122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:25:31.191175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:25:31.191253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:22:05.568191 (xid=0x2198x) >[2012-10-03 18:25:31.191274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:25:31.191290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:28:58.221149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:28:58.221206] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:28:58.221281] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:25:32.597841 (xid=0x2199x) >[2012-10-03 18:28:58.221303] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:28:58.221320] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:31:55.658289] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:31:55.661766] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:31:55.664151] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:32:25.256154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:32:25.256203] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:32:25.256267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:28:59.633387 (xid=0x2200x) >[2012-10-03 18:32:25.256288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:32:25.256304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:35:52.286113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:35:52.286163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:35:52.286228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:32:26.663291 (xid=0x2201x) >[2012-10-03 18:35:52.286263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:35:52.286279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:39:19.313110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:39:19.313160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:39:19.313226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:35:53.690781 (xid=0x2202x) >[2012-10-03 18:39:19.313247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:39:19.313264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:41:55.744384] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:41:55.747487] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:41:55.750044] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:42:46.343097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:42:46.343141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:42:46.343207] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:39:20.720525 (xid=0x2203x) >[2012-10-03 18:42:46.343229] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:42:46.343244] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:46:13.375104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:46:13.375154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:46:13.375223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:42:47.752513 (xid=0x2204x) >[2012-10-03 18:46:13.375244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:46:13.375272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:49:40.403187] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:49:40.403239] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:49:40.403307] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:46:14.780196 (xid=0x2205x) >[2012-10-03 18:49:40.403328] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:49:40.403345] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:51:56.126821] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:51:56.130333] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:51:56.132972] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 18:53:07.727129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:53:07.727176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:53:07.727242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:49:42.104011 (xid=0x2206x) >[2012-10-03 18:53:07.727262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:53:07.727278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 18:56:34.761111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 18:56:34.761164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 18:56:34.761231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:53:09.137885 (xid=0x2207x) >[2012-10-03 18:56:34.761253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 18:56:34.761270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:00:01.792118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:00:01.792176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:00:01.792265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 18:56:36.168894 (xid=0x2208x) >[2012-10-03 19:00:01.792287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:00:01.792303] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:01:56.223842] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:01:56.227371] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:01:56.230190] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:03:28.827111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:03:28.827158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:03:28.827223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:00:03.204155 (xid=0x2209x) >[2012-10-03 19:03:28.827244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:03:28.827260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:06:55.860106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:06:55.860156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:06:55.860222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:03:30.237670 (xid=0x2210x) >[2012-10-03 19:06:55.860243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:06:55.860258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:10:22.891113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:10:22.891163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:10:22.891230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:06:57.268579 (xid=0x2211x) >[2012-10-03 19:10:22.891262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:10:22.891279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:11:56.319498] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:11:56.322841] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:11:56.325554] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:13:49.925109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:13:49.925154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:13:49.925218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:10:24.302228 (xid=0x2212x) >[2012-10-03 19:13:49.925239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:13:49.925254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:17:16.959106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:17:16.959156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:17:16.959221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:13:51.336214 (xid=0x2213x) >[2012-10-03 19:17:16.959242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:17:16.959258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:20:43.990098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:20:43.990149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:20:43.990215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:17:18.366871 (xid=0x2214x) >[2012-10-03 19:20:43.990237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:20:43.990253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:21:56.704348] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:21:56.707813] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:21:56.711275] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:24:11.313111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:24:11.313164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:24:11.313239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:20:45.690648 (xid=0x2215x) >[2012-10-03 19:24:11.313261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:24:11.313279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:27:38.346111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:27:38.346160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:27:38.346226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:24:12.723631 (xid=0x2216x) >[2012-10-03 19:27:38.346248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:27:38.346264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:31:05.376099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:31:05.376149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:31:05.376215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:27:39.753398 (xid=0x2217x) >[2012-10-03 19:31:05.376236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:31:05.376253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:31:57.089389] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:31:57.092999] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:31:57.096160] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:34:32.701112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:34:32.701162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:34:32.701229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:31:07.077868 (xid=0x2218x) >[2012-10-03 19:34:32.701250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:34:32.701266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:37:59.732137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:37:59.732187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:37:59.732255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:34:34.109419 (xid=0x2219x) >[2012-10-03 19:37:59.732277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:37:59.732293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:41:26.760112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:41:26.760162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:41:26.760228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:38:01.137629 (xid=0x2220x) >[2012-10-03 19:41:26.760249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:41:26.760265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:41:57.175967] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:41:57.179181] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:41:57.181980] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:44:53.791100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:44:53.791151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:44:53.791216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:41:28.167839 (xid=0x2221x) >[2012-10-03 19:44:53.791237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:44:53.791253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:48:20.821124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:48:20.821175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:48:20.821242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:44:55.198673 (xid=0x2222x) >[2012-10-03 19:48:20.821264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:48:20.821281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:51:47.853119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:51:47.853170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:51:47.853237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:48:22.230379 (xid=0x2223x) >[2012-10-03 19:51:47.853259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:51:47.853275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:51:57.265897] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:51:57.269205] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:51:57.272067] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 19:55:14.883119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:55:14.883192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:55:14.883267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:51:49.260496 (xid=0x2224x) >[2012-10-03 19:55:14.883288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:55:14.883305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 19:58:41.912115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 19:58:41.912166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 19:58:41.912229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:55:16.289477 (xid=0x2225x) >[2012-10-03 19:58:41.912251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 19:58:41.912267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:01:57.344714] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:01:57.347372] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:01:57.349613] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:02:08.941132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:02:08.941185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:02:08.941256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 19:58:43.318025 (xid=0x2226x) >[2012-10-03 20:02:08.941278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:02:08.941295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:05:35.970129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:05:35.970180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:05:35.970248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:02:10.347287 (xid=0x2227x) >[2012-10-03 20:05:35.970280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:05:35.970298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:09:02.998155] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:09:02.998207] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:09:02.998274] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:05:37.375163 (xid=0x2228x) >[2012-10-03 20:09:02.998295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:09:02.998310] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:11:57.425934] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:11:57.429025] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:11:57.431651] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:12:30.025117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:12:30.025169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:12:30.025236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:09:04.401895 (xid=0x2229x) >[2012-10-03 20:12:30.025258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:12:30.025274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:15:57.056142] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:15:57.056193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:15:57.056260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:12:31.433041 (xid=0x2230x) >[2012-10-03 20:15:57.056281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:15:57.056312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:19:24.084119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:19:24.084172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:19:24.084236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:15:58.460967 (xid=0x2231x) >[2012-10-03 20:19:24.084258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:19:24.084275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:21:57.511232] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:21:57.514986] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:21:57.517916] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:22:51.112128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:22:51.112178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:22:51.112246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:19:25.488844 (xid=0x2232x) >[2012-10-03 20:22:51.112267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:22:51.112284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:26:18.144146] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:26:18.144197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:26:18.144264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:22:52.521070 (xid=0x2233x) >[2012-10-03 20:26:18.144286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:26:18.144302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:29:45.171163] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:29:45.171212] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:29:45.171290] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:26:19.548663 (xid=0x2234x) >[2012-10-03 20:29:45.171312] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:29:45.171327] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:31:57.595886] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:31:57.598545] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:31:57.601072] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:33:12.199143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:33:12.199199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:33:12.199278] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:29:46.576589 (xid=0x2235x) >[2012-10-03 20:33:12.199300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:33:12.199318] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:36:39.230165] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:36:39.230216] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:36:39.230283] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:33:13.607303 (xid=0x2236x) >[2012-10-03 20:36:39.230304] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:36:39.230320] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:40:06.257163] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:40:06.257216] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:40:06.257293] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:36:40.634521 (xid=0x2237x) >[2012-10-03 20:40:06.257328] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:40:06.257347] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:41:57.679037] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:41:57.682183] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:41:57.684984] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:43:33.285112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:43:33.285165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:43:33.285239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:40:07.661830 (xid=0x2238x) >[2012-10-03 20:43:33.286208] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:43:33.286249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:47:00.313158] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:47:00.313208] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:47:00.313274] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:43:34.690358 (xid=0x2239x) >[2012-10-03 20:47:00.313296] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:47:00.313313] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:50:27.340162] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:50:27.340213] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:50:27.340280] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:47:01.717675 (xid=0x2240x) >[2012-10-03 20:50:27.340301] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:50:27.340328] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:51:57.760132] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:51:57.763269] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:51:57.765591] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 20:53:54.368112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:53:54.368165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:53:54.368233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:50:28.745319 (xid=0x2241x) >[2012-10-03 20:53:54.368254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:53:54.368271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 20:57:21.397105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 20:57:21.397156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 20:57:21.397224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:53:55.774025 (xid=0x2242x) >[2012-10-03 20:57:21.397246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 20:57:21.397262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:00:48.424099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:00:48.424150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:00:48.424216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 20:57:22.801280 (xid=0x2243x) >[2012-10-03 21:00:48.424238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:00:48.424254] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:01:57.843970] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:01:57.846647] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:01:57.849172] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:04:15.455103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:04:15.455154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:04:15.455224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:00:49.832096 (xid=0x2244x) >[2012-10-03 21:04:15.455246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:04:15.455263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:07:42.777147] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:07:42.777197] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:07:42.777261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:04:17.154629 (xid=0x2245x) >[2012-10-03 21:07:42.777283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:07:42.777299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:11:09.808107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:11:09.808164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:11:09.808240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:07:44.185053 (xid=0x2246x) >[2012-10-03 21:11:09.808262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:11:09.808279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:11:58.227065] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:11:58.230414] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:11:58.233098] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:14:36.838103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:14:36.838158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:14:36.838228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:11:11.215454 (xid=0x2247x) >[2012-10-03 21:14:36.838250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:14:36.838266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:18:03.874124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:18:03.874173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:18:03.874240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:14:38.251111 (xid=0x2248x) >[2012-10-03 21:18:03.874261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:18:03.874278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:21:30.904101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:21:30.904151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:21:30.904218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:18:05.281094 (xid=0x2249x) >[2012-10-03 21:21:30.904239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:21:30.904255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:21:58.318601] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:21:58.321347] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:21:58.323712] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:24:57.934128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:24:57.934188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:24:57.934254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:21:32.310913 (xid=0x2250x) >[2012-10-03 21:24:57.934275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:24:57.934290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:28:25.261113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:28:25.261162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:28:25.261230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:24:59.638248 (xid=0x2251x) >[2012-10-03 21:28:25.261252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:28:25.261268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:31:52.292127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:31:52.292173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:31:52.292239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:28:26.669271 (xid=0x2252x) >[2012-10-03 21:31:52.292260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:31:52.292276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:31:58.704606] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:31:58.707393] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:31:58.709464] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:35:19.322121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:35:19.322178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:35:19.322253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:31:53.699718 (xid=0x2253x) >[2012-10-03 21:35:19.322286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:35:19.322304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:38:46.355117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:38:46.355169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:38:46.355236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:35:20.731981 (xid=0x2254x) >[2012-10-03 21:38:46.355257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:38:46.355274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:41:58.791177] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:41:58.794497] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:41:58.797337] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:42:13.385095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:42:13.385139] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:42:13.385205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:38:47.762120 (xid=0x2255x) >[2012-10-03 21:42:13.385225] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:42:13.385241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:45:40.417112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:45:40.417161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:45:40.417226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:42:14.794924 (xid=0x2256x) >[2012-10-03 21:45:40.417258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:45:40.417275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:49:07.449116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:49:07.449170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:49:07.449239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:45:41.826031 (xid=0x2257x) >[2012-10-03 21:49:07.449260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:49:07.449277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:51:58.884037] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:51:58.887593] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:51:58.890296] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 21:52:34.479094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:52:34.479140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:52:34.479203] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:49:08.856696 (xid=0x2258x) >[2012-10-03 21:52:34.479223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:52:34.479239] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:56:01.513111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:56:01.513161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:56:01.513226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:52:35.890359 (xid=0x2259x) >[2012-10-03 21:56:01.513248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:56:01.513264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 21:59:28.544073] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 21:59:28.544132] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 21:59:28.544199] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:56:02.920914 (xid=0x2260x) >[2012-10-03 21:59:28.544221] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 21:59:28.544236] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:01:58.976441] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:01:58.980198] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:01:58.983006] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:02:55.575113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:02:55.575160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:02:55.575226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 21:59:29.952025 (xid=0x2261x) >[2012-10-03 22:02:55.575248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:02:55.575264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:06:22.608057] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:06:22.608109] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:06:22.608175] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:02:56.985787 (xid=0x2262x) >[2012-10-03 22:06:22.608197] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:06:22.608213] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:09:49.640121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:09:49.640170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:09:49.640238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:06:24.017105 (xid=0x2263x) >[2012-10-03 22:09:49.640270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:09:49.640287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:11:59.070273] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:11:59.073439] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:11:59.075744] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:13:16.671128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:13:16.671174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:13:16.671240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:09:51.048263 (xid=0x2264x) >[2012-10-03 22:13:16.671261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:13:16.671277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:16:43.706103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:16:43.706151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:16:43.706219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:13:18.083068 (xid=0x2265x) >[2012-10-03 22:16:43.706240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:16:43.706257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:20:10.740135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:20:10.740184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:20:10.740253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:16:45.117472 (xid=0x2266x) >[2012-10-03 22:20:10.740275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:20:10.740304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:21:59.170340] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:21:59.173765] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:21:59.176331] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:23:37.774125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:23:37.774173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:23:37.774239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:20:12.150842 (xid=0x2267x) >[2012-10-03 22:23:37.774260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:23:37.774276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:27:04.809132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:27:04.809193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:27:04.809268] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:23:39.186105 (xid=0x2268x) >[2012-10-03 22:27:04.809290] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:27:04.809308] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:30:32.137092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:30:32.137144] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:30:32.137212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:27:06.513904 (xid=0x2269x) >[2012-10-03 22:30:32.137234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:30:32.137251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:31:59.561446] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:31:59.564724] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:31:59.567157] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:33:59.168117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:33:59.168170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:33:59.168245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:30:33.545134 (xid=0x2270x) >[2012-10-03 22:33:59.169003] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:33:59.169040] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:37:26.201115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:37:26.201168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:37:26.201235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:34:00.578413 (xid=0x2271x) >[2012-10-03 22:37:26.201257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:37:26.201273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:40:53.234075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:40:53.234125] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:40:53.234190] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:37:27.611148 (xid=0x2272x) >[2012-10-03 22:40:53.234213] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:40:53.234229] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:41:59.650820] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:41:59.653811] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:41:59.656849] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:44:20.262116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:44:20.262163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:44:20.262230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:40:54.639080 (xid=0x2273x) >[2012-10-03 22:44:20.262251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:44:20.262267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:47:47.292126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:47:47.292180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:47:47.292247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:44:21.668893 (xid=0x2274x) >[2012-10-03 22:47:47.292268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:47:47.292285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:51:14.614131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:51:14.614189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:51:14.614265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:47:48.991616 (xid=0x2275x) >[2012-10-03 22:51:14.614287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:51:14.614305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:52:00.030647] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:52:00.033326] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:52:00.035776] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 22:54:41.644111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:54:41.644164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:54:41.644231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:51:16.021096 (xid=0x2276x) >[2012-10-03 22:54:41.644253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:54:41.644269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 22:58:08.675115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 22:58:08.675176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 22:58:08.675251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:54:43.052070 (xid=0x2277x) >[2012-10-03 22:58:08.675273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 22:58:08.675290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:01:35.706080] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:01:35.706134] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:01:35.706205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 22:58:10.083083 (xid=0x2278x) >[2012-10-03 23:01:35.706228] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:01:35.706244] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:02:00.120336] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:02:00.123583] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:02:00.125875] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:05:02.736113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:05:02.736164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:05:02.736245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:01:37.112975 (xid=0x2279x) >[2012-10-03 23:05:02.736266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:05:02.736282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:08:29.766103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:08:29.766150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:08:29.766218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:05:04.143636 (xid=0x2280x) >[2012-10-03 23:08:29.766239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:08:29.766255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:11:56.797111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:11:56.797160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:11:56.797228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:08:31.174409 (xid=0x2281x) >[2012-10-03 23:11:56.797249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:11:56.797265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:12:00.208562] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:12:00.211803] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:12:00.214453] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:15:23.827115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:15:23.827167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:15:23.827235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:11:58.203969 (xid=0x2282x) >[2012-10-03 23:15:23.827267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:15:23.827284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:18:50.862113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:18:50.862164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:18:50.862231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:15:25.239106 (xid=0x2283x) >[2012-10-03 23:18:50.862252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:18:50.862269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:22:00.297755] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:22:00.301460] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:22:00.304535] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:22:17.893134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:22:17.893191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:22:17.893265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:18:52.269815 (xid=0x2284x) >[2012-10-03 23:22:17.893287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:22:17.893304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:25:44.923121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:25:44.923172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:25:44.923240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:22:19.300541 (xid=0x2285x) >[2012-10-03 23:25:44.923262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:25:44.923278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:29:11.953106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:29:11.953176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:29:11.953251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:25:46.329906 (xid=0x2286x) >[2012-10-03 23:29:11.953272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:29:11.953289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:32:00.384330] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:32:00.387362] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:32:00.389704] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:32:38.983134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:32:38.983183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:32:38.983252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:29:13.360082 (xid=0x2287x) >[2012-10-03 23:32:38.983273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:32:38.983290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:36:06.012156] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:36:06.012210] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:36:06.012277] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:32:40.389609 (xid=0x2288x) >[2012-10-03 23:36:06.012298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:36:06.012315] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:39:33.041123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:39:33.041174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:39:33.041242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:36:07.417870 (xid=0x2289x) >[2012-10-03 23:39:33.041275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:39:33.041292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:42:00.470561] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:42:00.473557] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:42:00.476300] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:43:00.071128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:43:00.071178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:43:00.071256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:39:34.448527 (xid=0x2290x) >[2012-10-03 23:43:00.071284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:43:00.071301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:46:27.105108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:46:27.105156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:46:27.105225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:43:01.481880 (xid=0x2291x) >[2012-10-03 23:46:27.105246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:46:27.105262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:49:54.133107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:49:54.133157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:49:54.133223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:46:28.511017 (xid=0x2292x) >[2012-10-03 23:49:54.133257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:49:54.133274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:52:00.561256] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:52:00.563836] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:52:00.566079] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-03 23:53:21.164081] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:53:21.164132] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:53:21.164205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:49:55.541674 (xid=0x2293x) >[2012-10-03 23:53:21.164226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:53:21.164244] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-03 23:56:48.195105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-03 23:56:48.195157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-03 23:56:48.195223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:53:22.571973 (xid=0x2294x) >[2012-10-03 23:56:48.195245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-03 23:56:48.195261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:00:15.223111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:00:15.223164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:00:15.223238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-03 23:56:49.600673 (xid=0x2295x) >[2012-10-04 00:00:15.223261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:00:15.223278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:02:00.644694] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:02:00.647932] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:02:00.650495] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:03:42.251104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:03:42.251156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:03:42.251224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:00:16.628206 (xid=0x2296x) >[2012-10-04 00:03:42.251245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:03:42.251261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:07:09.280082] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:07:09.280132] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:07:09.280199] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:03:43.657155 (xid=0x2297x) >[2012-10-04 00:07:09.280220] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:07:09.280237] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:10:36.307121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:10:36.307172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:10:36.307239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:07:10.684607 (xid=0x2298x) >[2012-10-04 00:10:36.307260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:10:36.307277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:12:00.729115] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:12:00.732318] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:12:00.735204] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:14:03.337114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:14:03.337165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:14:03.337231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:10:37.714239 (xid=0x2299x) >[2012-10-04 00:14:03.337253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:14:03.337270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:17:30.367107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:17:30.367159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:17:30.367226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:14:04.744100 (xid=0x2300x) >[2012-10-04 00:17:30.367247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:17:30.367264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:20:57.394104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:20:57.394152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:20:57.394218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:17:31.771732 (xid=0x2301x) >[2012-10-04 00:20:57.394240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:20:57.394257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:22:00.814355] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:22:00.817768] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:22:00.820700] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:24:24.425116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:24:24.425171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:24:24.425247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:20:58.802163 (xid=0x2302x) >[2012-10-04 00:24:24.425269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:24:24.425286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:27:51.455103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:27:51.455151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:27:51.455218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:24:25.832448 (xid=0x2303x) >[2012-10-04 00:27:51.455239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:27:51.455256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:31:18.483079] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:31:18.483142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:31:18.483221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:27:52.860684 (xid=0x2304x) >[2012-10-04 00:31:18.483243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:31:18.483261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:32:01.192839] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:32:01.195583] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:32:01.197962] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:34:45.806102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:34:45.806150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:34:45.806230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:31:20.183797 (xid=0x2305x) >[2012-10-04 00:34:45.806251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:34:45.806267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:38:12.835109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:38:12.835166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:38:12.835242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:34:47.212787 (xid=0x2306x) >[2012-10-04 00:38:12.835264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:38:12.835281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:41:39.865117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:41:39.865166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:41:39.865234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:38:14.242791 (xid=0x2307x) >[2012-10-04 00:41:39.865256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:41:39.865272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:42:01.575501] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:42:01.578321] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:42:01.580605] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:45:07.191111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:45:07.191164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:45:07.191231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:41:41.568178 (xid=0x2308x) >[2012-10-04 00:45:07.191265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:45:07.191283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:48:34.221104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:48:34.221160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:48:34.221234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:45:08.598213 (xid=0x2309x) >[2012-10-04 00:48:34.222003] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:48:34.222040] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:52:01.250127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:52:01.250183] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:52:01.250263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:48:35.627726 (xid=0x2310x) >[2012-10-04 00:52:01.251537] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:52:01.251581] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:52:01.659670] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:52:01.663080] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:52:01.666349] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 00:55:28.285127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:55:28.285177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:55:28.285245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:52:02.662691 (xid=0x2311x) >[2012-10-04 00:55:28.285267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:55:28.285283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 00:58:55.315126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 00:58:55.315179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 00:58:55.315246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:55:29.692060 (xid=0x2312x) >[2012-10-04 00:58:55.315268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 00:58:55.315284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:02:01.750115] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:02:01.753328] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:02:01.755796] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:02:22.345122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:02:22.345164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:02:22.345230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 00:58:56.722409 (xid=0x2313x) >[2012-10-04 01:02:22.345250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:02:22.345266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:05:49.380101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:05:49.380151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:05:49.380219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:02:23.756359 (xid=0x2314x) >[2012-10-04 01:05:49.380240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:05:49.380256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:09:16.409118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:09:16.409174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:09:16.409261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:05:50.786062 (xid=0x2315x) >[2012-10-04 01:09:16.409284] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:09:16.409301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:12:01.841735] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:12:01.844691] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:12:01.847215] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:12:43.439101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:12:43.439149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:12:43.439213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:09:17.816140 (xid=0x2316x) >[2012-10-04 01:12:43.439234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:12:43.439250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:16:10.472119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:16:10.472173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:16:10.472241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:12:44.849261 (xid=0x2317x) >[2012-10-04 01:16:10.472263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:16:10.472279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:19:37.503108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:19:37.503163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:19:37.503229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:16:11.880330 (xid=0x2318x) >[2012-10-04 01:19:37.503262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:19:37.503279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:22:01.934764] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:22:01.937905] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:22:01.940796] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:23:04.534101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:23:04.534149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:23:04.534215] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:19:38.910818 (xid=0x2319x) >[2012-10-04 01:23:04.534235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:23:04.534251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:26:31.569141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:26:31.569191] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:26:31.569259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:23:05.946032 (xid=0x2320x) >[2012-10-04 01:26:31.569280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:26:31.569297] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:29:58.600132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:29:58.600184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:29:58.600251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:26:32.977039 (xid=0x2321x) >[2012-10-04 01:29:58.600272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:29:58.600289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:32:02.030074] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:32:02.032921] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:32:02.035815] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:33:25.631024] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:33:25.631080] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:33:25.631154] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:30:00.008294 (xid=0x2322x) >[2012-10-04 01:33:25.631177] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:33:25.631196] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:36:52.665106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:36:52.665165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:36:52.665233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:33:27.042566 (xid=0x2323x) >[2012-10-04 01:36:52.665255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:36:52.665272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:40:19.698115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:40:19.698164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:40:19.698236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:36:54.075033 (xid=0x2324x) >[2012-10-04 01:40:19.698257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:40:19.698274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:42:02.126905] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:42:02.129632] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:42:02.132070] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:43:46.730109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:43:46.730162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:43:46.730228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:40:21.107746 (xid=0x2325x) >[2012-10-04 01:43:46.730249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:43:46.730265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:47:13.765106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:47:13.765166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:47:13.765241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:43:48.142397 (xid=0x2326x) >[2012-10-04 01:47:13.765263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:47:13.765281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:50:40.797064] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:50:40.797113] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:50:40.797180] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:47:15.174178 (xid=0x2327x) >[2012-10-04 01:50:40.797202] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:50:40.797218] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:52:02.221042] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:52:02.224065] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:52:02.226875] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 01:54:07.828099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:54:07.828146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:54:07.828213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:50:42.205551 (xid=0x2328x) >[2012-10-04 01:54:07.828233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:54:07.828251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 01:57:34.863112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 01:57:34.863167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 01:57:34.863237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:54:09.240016 (xid=0x2329x) >[2012-10-04 01:57:34.863260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 01:57:34.863276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:01:01.894105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:01:01.894151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:01:01.894221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 01:57:36.271315 (xid=0x2330x) >[2012-10-04 02:01:01.894243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:01:01.894259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:02:02.316270] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:02:02.319594] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:02:02.322845] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:04:28.926117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:04:28.926174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:04:28.926265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:01:03.303625 (xid=0x2331x) >[2012-10-04 02:04:28.926287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:04:28.926304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:07:55.960106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:07:55.960158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:07:55.960224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:04:30.337039 (xid=0x2332x) >[2012-10-04 02:07:55.960246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:07:55.960263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:11:22.990121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:11:22.990184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:11:22.990259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:07:57.367517 (xid=0x2333x) >[2012-10-04 02:11:22.990281] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:11:22.990298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:12:02.407438] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:12:02.410726] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:12:02.413422] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:14:50.020110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:14:50.020158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:14:50.020224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:11:24.397589 (xid=0x2334x) >[2012-10-04 02:14:50.020258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:14:50.020276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:18:17.054110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:18:17.054167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:18:17.054232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:14:51.430926 (xid=0x2335x) >[2012-10-04 02:18:17.054253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:18:17.054269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:21:44.083147] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:21:44.083196] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:21:44.083270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:18:18.460778 (xid=0x2336x) >[2012-10-04 02:21:44.084069] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:21:44.084113] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:22:02.497136] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:22:02.499760] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:22:02.502047] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:25:11.113127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:25:11.113174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:25:11.113240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:21:45.490619 (xid=0x2337x) >[2012-10-04 02:25:11.113261] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:25:11.113288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:28:38.145121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:28:38.145170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:28:38.145234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:25:12.522409 (xid=0x2338x) >[2012-10-04 02:28:38.145255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:28:38.145271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:32:02.582999] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:32:02.586301] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:32:02.589019] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:32:05.175107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:32:05.175144] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:32:05.175206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:28:39.552026 (xid=0x2339x) >[2012-10-04 02:32:05.175226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:32:05.175242] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:35:32.207099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:35:32.207145] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:35:32.207212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:32:06.584269 (xid=0x2340x) >[2012-10-04 02:35:32.207234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:35:32.207250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:38:59.533129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:38:59.533178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:38:59.533256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:35:33.910029 (xid=0x2341x) >[2012-10-04 02:38:59.533276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:38:59.533292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:42:02.964703] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:42:02.967524] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:42:02.970329] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:42:26.561104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:42:26.561149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:42:26.561214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:39:00.938619 (xid=0x2342x) >[2012-10-04 02:42:26.561234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:42:26.561250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:45:53.592115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:45:53.592165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:45:53.592232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:42:27.969005 (xid=0x2343x) >[2012-10-04 02:45:53.592254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:45:53.592271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:49:20.622161] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:49:20.622215] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:49:20.622288] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:45:54.999656 (xid=0x2344x) >[2012-10-04 02:49:20.622323] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:49:20.622342] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:52:03.056313] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:52:03.059426] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:52:03.061734] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 02:52:47.653138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:52:47.653186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:52:47.653251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:49:22.030392 (xid=0x2345x) >[2012-10-04 02:52:47.653272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:52:47.653288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:56:14.686129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:56:14.686187] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:56:14.686261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:52:49.063817 (xid=0x2346x) >[2012-10-04 02:56:14.686283] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:56:14.686300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 02:59:41.718752] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 02:59:41.718802] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 02:59:41.718874] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:56:16.095161 (xid=0x2347x) >[2012-10-04 02:59:41.718895] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 02:59:41.718911] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:02:03.147774] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:02:03.150929] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:02:03.153334] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:03:08.748127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:03:08.748175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:03:08.748242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 02:59:43.124911 (xid=0x2348x) >[2012-10-04 03:03:08.748263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:03:08.748278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:06:35.781129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:06:35.781178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:06:35.781246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:03:10.158432 (xid=0x2349x) >[2012-10-04 03:06:35.781268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:06:35.781284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:10:02.813115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:10:02.813170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:10:02.813242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:06:37.190149 (xid=0x2350x) >[2012-10-04 03:10:02.813264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:10:02.813282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:12:03.239052] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:12:03.242378] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:12:03.245152] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:13:29.843123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:13:29.843172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:13:29.843239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:10:04.219867 (xid=0x2351x) >[2012-10-04 03:13:29.843260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:13:29.843277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:16:56.874112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:16:56.874162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:16:56.874228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:13:31.251633 (xid=0x2352x) >[2012-10-04 03:16:56.874250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:16:56.874266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:20:23.905141] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:20:23.905193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:20:23.905260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:16:58.282516 (xid=0x2353x) >[2012-10-04 03:20:23.905280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:20:23.905296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:22:03.328182] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:22:03.330916] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:22:03.333255] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:23:50.934112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:23:50.934160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:23:50.934225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:20:25.311660 (xid=0x2354x) >[2012-10-04 03:23:50.934245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:23:50.934261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:27:17.966123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:27:17.966173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:27:17.966237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:23:52.343542 (xid=0x2355x) >[2012-10-04 03:27:17.966259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:27:17.966275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:30:44.998144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:30:44.998195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:30:44.998261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:27:19.375253 (xid=0x2356x) >[2012-10-04 03:30:44.998282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:30:44.998299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:32:03.417234] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:32:03.420202] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:32:03.422398] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:34:12.026070] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:34:12.026129] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:34:12.026196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:30:46.403672 (xid=0x2357x) >[2012-10-04 03:34:12.026216] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:34:12.026231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:37:39.057149] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:37:39.057205] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:37:39.057275] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:34:13.434576 (xid=0x2358x) >[2012-10-04 03:37:39.057298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:37:39.057315] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:41:06.381116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:41:06.381167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:41:06.381235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:37:40.758322 (xid=0x2359x) >[2012-10-04 03:41:06.381257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:41:06.381273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:42:03.797843] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:42:03.801136] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:42:03.803685] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:44:33.409110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:44:33.409157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:44:33.409224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:41:07.786431 (xid=0x2360x) >[2012-10-04 03:44:33.409255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:44:33.409272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:48:00.440111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:48:00.440160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:48:00.440227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:44:34.817418 (xid=0x2361x) >[2012-10-04 03:48:00.440249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:48:00.440265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:51:27.468098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:51:27.468151] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:51:27.468220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:48:01.845693 (xid=0x2362x) >[2012-10-04 03:51:27.468242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:51:27.468259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:52:03.883711] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:52:03.887081] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:52:03.889517] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 03:54:54.497096] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:54:54.497147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:54:54.497214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:51:28.874324 (xid=0x2363x) >[2012-10-04 03:54:54.497236] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:54:54.497266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 03:58:21.528107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 03:58:21.528166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 03:58:21.528240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:54:55.905595 (xid=0x2364x) >[2012-10-04 03:58:21.528263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 03:58:21.528281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:01:48.556080] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:01:48.556131] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:01:48.556199] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 03:58:22.933555 (xid=0x2365x) >[2012-10-04 04:01:48.556221] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:01:48.556238] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:02:03.967990] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:02:03.971498] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:02:03.974483] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:05:15.584097] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:05:15.584149] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:05:15.584217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:01:49.961614 (xid=0x2366x) >[2012-10-04 04:05:15.584238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:05:15.584255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:08:42.615112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:08:42.615162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:08:42.615245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:05:16.992311 (xid=0x2367x) >[2012-10-04 04:08:42.615266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:08:42.615283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:12:04.344686] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:12:04.348149] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:12:04.350627] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:12:09.937124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:12:09.937170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:12:09.937234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:08:44.314146 (xid=0x2368x) >[2012-10-04 04:12:09.937255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:12:09.937271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:15:36.970134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:15:36.970186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:15:36.970254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:12:11.347190 (xid=0x2369x) >[2012-10-04 04:15:36.970276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:15:36.970293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:19:03.999144] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:19:03.999193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:19:03.999259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:15:38.376664 (xid=0x2370x) >[2012-10-04 04:19:03.999295] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:19:03.999312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:22:04.731059] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:22:04.733480] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:22:04.735591] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:22:31.329111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:22:31.329163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:22:31.329239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:19:05.706499 (xid=0x2371x) >[2012-10-04 04:22:31.330219] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:22:31.330262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:25:58.360129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:25:58.360178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:25:58.360245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:22:32.737066 (xid=0x2372x) >[2012-10-04 04:25:58.360266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:25:58.360282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:29:25.387101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:29:25.387155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:29:25.387223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:25:59.764562 (xid=0x2373x) >[2012-10-04 04:29:25.387244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:29:25.387272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:32:04.817030] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:32:04.820523] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:32:04.823212] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:32:52.417127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:32:52.417176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:32:52.417253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:29:26.794098 (xid=0x2374x) >[2012-10-04 04:32:52.417275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:32:52.417293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:36:19.451115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:36:19.451167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:36:19.451233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:32:53.828377 (xid=0x2375x) >[2012-10-04 04:36:19.451255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:36:19.451271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:39:46.479113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:39:46.479167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:39:46.479237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:36:20.856248 (xid=0x2376x) >[2012-10-04 04:39:46.479258] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:39:46.479275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:42:04.907444] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:42:04.910836] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:42:04.913398] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:43:13.509110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:43:13.509164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:43:13.509231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:39:47.886505 (xid=0x2377x) >[2012-10-04 04:43:13.509251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:43:13.509267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:46:40.540123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:46:40.540175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:46:40.540243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:43:14.917367 (xid=0x2378x) >[2012-10-04 04:46:40.540264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:46:40.540281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:50:07.568122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:50:07.568171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:50:07.568238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:46:41.945026 (xid=0x2379x) >[2012-10-04 04:50:07.568259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:50:07.568275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:52:04.992973] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:52:04.995878] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:52:04.998387] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 04:53:34.597113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:53:34.597168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:53:34.597244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:50:08.974647 (xid=0x2380x) >[2012-10-04 04:53:34.598174] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:53:34.598218] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 04:57:01.627116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 04:57:01.627167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 04:57:01.627233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:53:36.003863 (xid=0x2381x) >[2012-10-04 04:57:01.627254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 04:57:01.627271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:00:28.654122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:00:28.654174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:00:28.654241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 04:57:03.031638 (xid=0x2382x) >[2012-10-04 05:00:28.654262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:00:28.654278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:02:05.076642] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:02:05.079957] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:02:05.082892] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:03:55.683122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:03:55.683182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:03:55.683248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:00:30.060801 (xid=0x2383x) >[2012-10-04 05:03:55.683269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:03:55.683284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:07:22.713115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:07:22.713174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:07:22.713248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:03:57.089836 (xid=0x2384x) >[2012-10-04 05:07:22.713270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:07:22.713287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:10:49.740046] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:10:49.740096] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:10:49.740172] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:07:24.117410 (xid=0x2385x) >[2012-10-04 05:10:49.740193] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:10:49.740209] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:12:05.160665] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:12:05.163443] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:12:05.165672] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:14:16.769108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:14:16.769160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:14:16.769226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:10:51.145971 (xid=0x2386x) >[2012-10-04 05:14:16.769259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:14:16.769276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:17:43.805123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:17:43.805177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:17:43.805246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:14:18.182105 (xid=0x2387x) >[2012-10-04 05:17:43.805267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:17:43.805284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:21:10.832122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:21:10.832173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:21:10.832240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:17:45.209407 (xid=0x2388x) >[2012-10-04 05:21:10.832262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:21:10.832278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:22:05.249230] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:22:05.252544] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:22:05.255460] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:24:37.861101] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:24:37.861155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:24:37.861223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:21:12.238494 (xid=0x2389x) >[2012-10-04 05:24:37.861259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:24:37.861276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:28:04.890115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:28:04.890167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:28:04.890241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:24:39.267229 (xid=0x2390x) >[2012-10-04 05:28:04.891338] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:28:04.891382] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:31:31.917114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:31:31.917167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:31:31.917234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:28:06.294397 (xid=0x2391x) >[2012-10-04 05:31:31.917255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:31:31.917272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:32:05.332267] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:32:05.335243] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:32:05.337827] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:34:58.947121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:34:58.947173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:34:58.947241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:31:33.324127 (xid=0x2392x) >[2012-10-04 05:34:58.947262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:34:58.947279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:38:25.975131] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:38:25.975202] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:38:25.975269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:35:00.352742 (xid=0x2393x) >[2012-10-04 05:38:25.975290] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:38:25.975307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:41:53.004119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:41:53.004170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:41:53.004238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:38:27.381391 (xid=0x2394x) >[2012-10-04 05:41:53.004260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:41:53.004276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:42:05.420294] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:42:05.423533] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:42:05.425893] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:45:20.037125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:45:20.037177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:45:20.037243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:41:54.414373 (xid=0x2395x) >[2012-10-04 05:45:20.037264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:45:20.037280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:48:47.065112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:48:47.065164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:48:47.065229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:45:21.442537 (xid=0x2396x) >[2012-10-04 05:48:47.065263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:48:47.065280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:52:05.500638] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:52:05.503883] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:52:05.506860] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 05:52:14.095105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:52:14.095150] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:52:14.095214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:48:48.472043 (xid=0x2397x) >[2012-10-04 05:52:14.095235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:52:14.095252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:55:41.127107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:55:41.127161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:55:41.127228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:52:15.505129 (xid=0x2398x) >[2012-10-04 05:55:41.127249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:55:41.127266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 05:59:08.157117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 05:59:08.157177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 05:59:08.157251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:55:42.534332 (xid=0x2399x) >[2012-10-04 05:59:08.157998] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 05:59:08.158059] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:02:05.592035] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:02:05.595165] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:02:05.597422] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:02:35.187094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:02:35.187148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:02:35.187216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 05:59:09.564127 (xid=0x2400x) >[2012-10-04 06:02:35.187237] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:02:35.187253] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:06:02.222102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:06:02.222153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:06:02.222218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:02:36.599709 (xid=0x2401x) >[2012-10-04 06:06:02.222239] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:06:02.222256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:09:29.253111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:09:29.253162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:09:29.253227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:06:03.630300 (xid=0x2402x) >[2012-10-04 06:09:29.253248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:09:29.253264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:12:05.681928] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:12:05.685396] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:12:05.688092] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:12:56.282102] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:12:56.282154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:12:56.282220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:09:30.659071 (xid=0x2403x) >[2012-10-04 06:12:56.282242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:12:56.282258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:16:23.314070] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:16:23.314119] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:16:23.314186] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:12:57.692024 (xid=0x2404x) >[2012-10-04 06:16:23.314207] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:16:23.314223] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:19:50.342099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:19:50.342146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:19:50.342213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:16:24.718820 (xid=0x2405x) >[2012-10-04 06:19:50.342235] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:19:50.342251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:22:05.766672] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:22:05.770267] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:22:05.773535] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:23:17.369122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:23:17.369172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:23:17.369237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:19:51.746415 (xid=0x2406x) >[2012-10-04 06:23:17.369257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:23:17.369273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:26:44.401159] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:26:44.401207] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:26:44.401273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:23:18.778791 (xid=0x2407x) >[2012-10-04 06:26:44.401294] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:26:44.401311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:30:11.429115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:30:11.429163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:30:11.429229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:26:45.805912 (xid=0x2408x) >[2012-10-04 06:30:11.429250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:30:11.429267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:32:05.853081] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:32:05.856351] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:32:05.858902] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:33:38.457170] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:33:38.457219] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:33:38.457286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:30:12.834654 (xid=0x2409x) >[2012-10-04 06:33:38.457308] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:33:38.457324] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:37:05.488120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:37:05.488173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:37:05.488238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:33:39.866053 (xid=0x2410x) >[2012-10-04 06:37:05.488259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:37:05.488275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:40:32.517117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:40:32.517169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:40:32.517244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:37:06.895041 (xid=0x2411x) >[2012-10-04 06:40:32.517266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:40:32.517284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:42:05.941155] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:42:05.944556] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:42:05.947630] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:43:59.548115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:43:59.548166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:43:59.548246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:40:33.925299 (xid=0x2412x) >[2012-10-04 06:43:59.548268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:43:59.548283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:47:26.580105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:47:26.580163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:47:26.580238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:44:00.958014 (xid=0x2413x) >[2012-10-04 06:47:26.580260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:47:26.580277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:50:53.611115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:50:53.611166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:50:53.611234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:47:27.988100 (xid=0x2414x) >[2012-10-04 06:50:53.611255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:50:53.611272] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:52:06.032911] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:52:06.036394] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:52:06.039423] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 06:54:20.642154] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:54:20.642202] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:54:20.642266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:50:55.019483 (xid=0x2415x) >[2012-10-04 06:54:20.642298] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:54:20.642315] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 06:57:47.675143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 06:57:47.675195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 06:57:47.675261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:54:22.052140 (xid=0x2416x) >[2012-10-04 06:57:47.675282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 06:57:47.675299] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:01:14.706103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:01:14.706153] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:01:14.706226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 06:57:49.083060 (xid=0x2417x) >[2012-10-04 07:01:14.706248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:01:14.706264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:02:06.128233] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:02:06.131499] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:02:06.134201] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:04:41.739076] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:04:41.739126] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:04:41.739196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:01:16.116651 (xid=0x2418x) >[2012-10-04 07:04:41.739218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:04:41.739234] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:08:08.772085] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:08:08.772148] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:08:08.772213] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:04:43.149677 (xid=0x2419x) >[2012-10-04 07:08:08.772234] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:08:08.772250] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:11:35.804122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:11:35.804174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:11:35.804242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:08:10.181263 (xid=0x2420x) >[2012-10-04 07:11:35.804264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:11:35.804281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:12:06.221795] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:12:06.224972] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:12:06.227331] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:15:02.835116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:15:02.835165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:15:02.835231] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:11:37.212690 (xid=0x2421x) >[2012-10-04 07:15:02.835253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:15:02.835269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:18:29.869170] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:18:29.869224] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:18:29.869292] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:15:04.246244 (xid=0x2422x) >[2012-10-04 07:18:29.869326] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:18:29.869344] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:21:56.902111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:21:56.902160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:21:56.902225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:18:31.278895 (xid=0x2423x) >[2012-10-04 07:21:56.902247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:21:56.902265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:22:06.317863] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:22:06.321185] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:22:06.323950] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:25:23.933133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:25:23.933195] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:25:23.933270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:21:58.310627 (xid=0x2424x) >[2012-10-04 07:25:23.933291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:25:23.933309] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:28:51.261110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:28:51.261158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:28:51.261226] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:25:25.638274 (xid=0x2425x) >[2012-10-04 07:28:51.261259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:28:51.261277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:32:06.696329] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:32:06.698587] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:32:18.292133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:32:18.292181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:32:18.292245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:28:52.669177 (xid=0x2426x) >[2012-10-04 07:32:18.292266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:32:18.292281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:35:45.322127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:35:45.322177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:35:45.322244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:32:19.699261 (xid=0x2427x) >[2012-10-04 07:35:45.322266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:35:45.322282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:39:12.352129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:39:12.352178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:39:12.352246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:35:46.729475 (xid=0x2428x) >[2012-10-04 07:39:12.352267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:39:12.352283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:42:06.784168] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:42:06.787481] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:42:06.790325] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:42:39.381123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:42:39.381174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:42:39.381248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:39:13.758292 (xid=0x2429x) >[2012-10-04 07:42:39.381269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:42:39.381287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:46:06.412143] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:46:06.412193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:46:06.412260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:42:40.789290 (xid=0x2430x) >[2012-10-04 07:46:06.412282] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:46:06.412298] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:49:33.443126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:49:33.443180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:49:33.443254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:46:07.820488 (xid=0x2431x) >[2012-10-04 07:49:33.443276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:49:33.443293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:52:06.873818] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:52:06.876703] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:52:06.879098] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 07:53:00.473109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:53:00.473159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:53:00.473225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:49:34.850030 (xid=0x2432x) >[2012-10-04 07:53:00.473246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:53:00.473262] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:56:27.504106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:56:27.504155] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:56:27.504219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:53:01.881295 (xid=0x2433x) >[2012-10-04 07:56:27.504240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:56:27.504256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 07:59:54.538129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 07:59:54.538180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 07:59:54.538247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:56:28.915910 (xid=0x2434x) >[2012-10-04 07:59:54.538268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 07:59:54.538285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:02:06.967456] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:02:06.970526] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:02:06.972976] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:03:21.569119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:03:21.569166] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:03:21.569241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 07:59:55.946132 (xid=0x2435x) >[2012-10-04 08:03:21.569262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:03:21.569279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:06:48.600113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:06:48.600161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:06:48.600228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:03:22.977196 (xid=0x2436x) >[2012-10-04 08:06:48.600249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:06:48.600266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:10:15.632123] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:10:15.632173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:10:15.632243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:06:50.009411 (xid=0x2437x) >[2012-10-04 08:10:15.632265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:10:15.632282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:12:07.060050] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:12:07.062965] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:12:07.065136] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:13:42.663137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:13:42.663194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:13:42.663270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:10:17.039884 (xid=0x2438x) >[2012-10-04 08:13:42.663304] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:13:42.663322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:17:09.695136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:17:09.695186] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:17:09.695254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:13:44.072744 (xid=0x2439x) >[2012-10-04 08:17:09.695276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:17:09.695293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:20:36.729105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:20:36.729160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:20:36.729233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:17:11.106407 (xid=0x2440x) >[2012-10-04 08:20:36.729255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:20:36.729273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:22:07.152692] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:22:07.156328] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:22:07.159246] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:24:03.760120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:24:03.760167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:24:03.760234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:20:38.136838 (xid=0x2441x) >[2012-10-04 08:24:03.760255] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:24:03.760280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:27:30.791110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:27:30.791160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:27:30.791227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:24:05.167819 (xid=0x2442x) >[2012-10-04 08:27:30.791248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:27:30.791265] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:30:57.823108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:30:57.823158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:30:57.823224] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:27:32.199852 (xid=0x2443x) >[2012-10-04 08:30:57.823245] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:30:57.823261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:32:07.243227] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:32:07.246712] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:32:07.249668] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:34:24.853105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:34:24.853154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:34:24.853221] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:30:59.230009 (xid=0x2444x) >[2012-10-04 08:34:24.853242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:34:24.853259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:37:51.885114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:37:51.885164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:37:51.885241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:34:26.262286 (xid=0x2445x) >[2012-10-04 08:37:51.885262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:37:51.885278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:41:18.917113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:41:18.917164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:41:18.917232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:37:53.294325 (xid=0x2446x) >[2012-10-04 08:41:18.917254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:41:18.917271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:42:07.335679] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:42:07.338638] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:42:07.340996] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:44:45.948121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:44:45.948175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:44:45.948245] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:41:20.325048 (xid=0x2447x) >[2012-10-04 08:44:45.948267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:44:45.948283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:48:12.979117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:48:12.979165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:48:12.979233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:44:47.356169 (xid=0x2448x) >[2012-10-04 08:48:12.979265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:48:12.979282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:51:40.011105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:51:40.011167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:51:40.011247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:48:14.388259 (xid=0x2449x) >[2012-10-04 08:51:40.011270] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:51:40.011287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:52:07.425118] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:52:07.428195] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:52:07.430779] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 08:55:07.040075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:55:07.040124] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:55:07.040192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:51:41.417287 (xid=0x2450x) >[2012-10-04 08:55:07.040213] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:55:07.040230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 08:58:34.072110] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 08:58:34.072162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 08:58:34.072230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:55:08.449802 (xid=0x2451x) >[2012-10-04 08:58:34.072252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 08:58:34.072269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:02:01.104113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:02:01.104162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:02:01.104228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 08:58:35.481504 (xid=0x2452x) >[2012-10-04 09:02:01.104250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:02:01.104266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:02:07.517028] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:02:07.520321] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:02:07.522738] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:05:28.135111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:05:28.135161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:05:28.135228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:02:02.512022 (xid=0x2453x) >[2012-10-04 09:05:28.135250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:05:28.135266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:08:55.167129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:08:55.167179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:08:55.167244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:05:29.543964 (xid=0x2454x) >[2012-10-04 09:08:55.167266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:08:55.167282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:12:07.604423] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:12:07.608015] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:12:07.611086] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:12:22.199122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:12:22.199169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:12:22.199232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:08:56.575891 (xid=0x2455x) >[2012-10-04 09:12:22.199252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:12:22.199268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:15:49.231113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:15:49.231164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:15:49.231230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:12:23.608272 (xid=0x2456x) >[2012-10-04 09:15:49.231250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:15:49.231267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:19:16.263107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:19:16.263160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:19:16.263225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:15:50.640465 (xid=0x2457x) >[2012-10-04 09:19:16.263247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:19:16.263263] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:22:07.697513] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:22:07.700314] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:22:07.702419] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:22:43.295122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:22:43.295182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:22:43.295263] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:19:17.672703 (xid=0x2458x) >[2012-10-04 09:22:43.296262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:22:43.296306] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:26:10.327119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:26:10.327173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:26:10.327243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:22:44.704310 (xid=0x2459x) >[2012-10-04 09:26:10.328695] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:26:10.328735] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:29:37.356121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:29:37.356180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:29:37.356255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:26:11.733280 (xid=0x2460x) >[2012-10-04 09:29:37.356277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:29:37.356295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:32:07.788933] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:32:07.792620] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:32:07.795427] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:33:04.386139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:33:04.386199] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:33:04.386265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:29:38.762902 (xid=0x2461x) >[2012-10-04 09:33:04.386286] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:33:04.386301] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:36:31.423100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:36:31.423152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:36:31.423220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:33:05.800162 (xid=0x2462x) >[2012-10-04 09:36:31.423242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:36:31.423259] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:39:58.453179] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:39:58.453228] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:39:58.453295] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:36:32.830059 (xid=0x2463x) >[2012-10-04 09:39:58.453316] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:39:58.453333] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:42:07.881171] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:42:07.884755] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:42:07.887627] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:43:25.483112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:43:25.483162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:43:25.483229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:39:59.859970 (xid=0x2464x) >[2012-10-04 09:43:25.483263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:43:25.483280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:46:52.516119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:46:52.516171] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:46:52.516239] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:43:26.893021 (xid=0x2465x) >[2012-10-04 09:46:52.516260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:46:52.516277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:50:19.545119] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:50:19.545170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:50:19.545238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:46:53.922597 (xid=0x2466x) >[2012-10-04 09:50:19.545260] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:50:19.545276] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:52:07.970253] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:52:07.973687] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:52:07.976690] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 09:53:46.575095] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:53:46.575143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:53:46.575206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:50:20.952800 (xid=0x2467x) >[2012-10-04 09:53:46.575227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:53:46.575255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 09:57:13.607127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 09:57:13.607178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 09:57:13.607246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:53:47.984597 (xid=0x2468x) >[2012-10-04 09:57:13.607267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 09:57:13.607283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:00:40.641081] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:00:40.641142] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:00:40.641219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 09:57:15.018206 (xid=0x2469x) >[2012-10-04 10:00:40.641241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:00:40.641258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:02:08.064720] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:02:08.067775] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:02:08.070470] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:04:07.673106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:04:07.673154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:04:07.673220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:00:42.049895 (xid=0x2470x) >[2012-10-04 10:04:07.673241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:04:07.673256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:07:34.705108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:07:34.705167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:07:34.705257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:04:09.082336 (xid=0x2471x) >[2012-10-04 10:07:34.705280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:07:34.705296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:11:01.735070] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:11:01.735125] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:11:01.735191] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:07:36.112029 (xid=0x2472x) >[2012-10-04 10:11:01.735213] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:11:01.735230] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:12:08.156036] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:12:08.158960] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:12:08.161227] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:14:28.766128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:14:28.766179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:14:28.766246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:11:03.143782 (xid=0x2473x) >[2012-10-04 10:14:28.766267] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:14:28.766283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:17:55.799114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:17:55.799163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:17:55.799230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:14:30.175895 (xid=0x2474x) >[2012-10-04 10:17:55.799264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:17:55.799282] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:21:22.830109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:21:22.830162] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:21:22.830229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:17:57.207491 (xid=0x2475x) >[2012-10-04 10:21:22.830251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:21:22.830268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:22:08.250740] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:22:08.254171] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:22:08.256680] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:24:49.864100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:24:49.864152] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:24:49.864219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:21:24.241027 (xid=0x2476x) >[2012-10-04 10:24:49.864241] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:24:49.864257] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:28:16.896115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:28:16.896167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:28:16.896235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:24:51.273396 (xid=0x2477x) >[2012-10-04 10:28:16.896256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:28:16.896284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:31:43.926106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:31:43.926167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:31:43.926242] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:28:18.302854 (xid=0x2478x) >[2012-10-04 10:31:43.926264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:31:43.926280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:32:08.342521] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:32:08.346110] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:32:08.349233] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:35:10.957114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:35:10.957167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:35:10.957238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:31:45.334638 (xid=0x2479x) >[2012-10-04 10:35:10.958275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:35:10.958318] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:38:37.989120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:38:37.989179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:38:37.989254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:35:12.366505 (xid=0x2480x) >[2012-10-04 10:38:37.989275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:38:37.989292] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:42:05.018134] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:42:05.018185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:42:05.018266] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:38:39.395033 (xid=0x2481x) >[2012-10-04 10:42:05.018288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:42:05.018304] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:42:08.429001] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:42:08.431910] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:42:08.434419] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:45:32.047115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:45:32.047177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:45:32.047253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:42:06.424309 (xid=0x2482x) >[2012-10-04 10:45:32.047275] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:45:32.047293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:48:59.078124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:48:59.078174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:48:59.078241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:45:33.455213 (xid=0x2483x) >[2012-10-04 10:48:59.078262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:48:59.078279] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:52:08.511996] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:52:08.514403] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:52:08.516438] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 10:52:26.107125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:52:26.107172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:52:26.107241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:49:00.484213 (xid=0x2484x) >[2012-10-04 10:52:26.107262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:52:26.107277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:55:53.139137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:55:53.139192] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:55:53.139267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:52:27.516024 (xid=0x2485x) >[2012-10-04 10:55:53.140195] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:55:53.140238] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 10:59:20.167112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 10:59:20.167164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 10:59:20.167230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:55:54.544358 (xid=0x2486x) >[2012-10-04 10:59:20.167252] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 10:59:20.167269] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:02:08.598766] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:02:08.601978] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:02:08.604522] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:02:47.197130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:02:47.197189] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:02:47.197256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 10:59:21.574377 (xid=0x2487x) >[2012-10-04 11:02:47.197276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:02:47.197291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:06:14.229092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:06:14.229143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:06:14.229209] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:02:48.605202 (xid=0x2488x) >[2012-10-04 11:06:14.229231] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:06:14.229247] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:09:41.257155] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:09:41.257216] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:09:41.257292] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:06:15.634316 (xid=0x2489x) >[2012-10-04 11:09:41.257315] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:09:41.257332] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:12:08.683987] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:12:08.687052] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:12:08.689783] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:13:08.285125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:13:08.285175] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:13:08.285241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:09:42.662716 (xid=0x2490x) >[2012-10-04 11:13:08.285274] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:13:08.285291] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:16:35.315120] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:16:35.315180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:16:35.315255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:13:09.692019 (xid=0x2491x) >[2012-10-04 11:16:35.315277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:16:35.315294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:20:02.341984] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:20:02.342039] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:20:02.342121] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:16:36.719738 (xid=0x2492x) >[2012-10-04 11:20:02.342149] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:20:02.342171] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:22:08.765762] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:22:08.768526] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:22:08.770876] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:23:29.370117] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:23:29.370169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:23:29.370236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:20:03.747413 (xid=0x2493x) >[2012-10-04 11:23:29.370271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:23:29.370289] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:26:56.402122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:26:56.402173] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:26:56.402241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:23:30.778923 (xid=0x2494x) >[2012-10-04 11:26:56.402262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:26:56.402280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:30:23.430126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:30:23.430179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:30:23.430247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:26:57.807381 (xid=0x2495x) >[2012-10-04 11:30:23.430268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:30:23.430285] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:32:08.852562] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:32:08.855461] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:32:08.858332] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:33:50.459165] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:33:50.459216] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:33:50.459284] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:30:24.836041 (xid=0x2496x) >[2012-10-04 11:33:50.459306] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:33:50.459322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:37:17.798059] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:37:17.798120] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:37:17.798187] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:33:52.175480 (xid=0x2497x) >[2012-10-04 11:37:17.798208] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:37:17.798224] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:40:44.830098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:40:44.830159] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:40:44.830234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:37:19.207033 (xid=0x2498x) >[2012-10-04 11:40:44.830256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:40:44.830273] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:42:09.254217] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:42:09.257716] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:42:09.260599] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:44:11.862112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:44:11.862164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:44:11.862233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:40:46.239065 (xid=0x2499x) >[2012-10-04 11:44:11.862254] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:44:11.862270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:47:38.896106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:47:38.896167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:47:38.896240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:44:13.272885 (xid=0x2500x) >[2012-10-04 11:47:38.896277] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:47:38.896295] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:51:05.927115] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:51:05.927167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:51:05.927235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:47:40.304705 (xid=0x2501x) >[2012-10-04 11:51:05.927257] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:51:05.927274] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:52:09.349795] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:52:09.352480] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:52:09.354665] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 11:54:32.960124] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:54:32.960174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:54:32.960241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:51:07.337074 (xid=0x2502x) >[2012-10-04 11:54:32.960262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:54:32.960278] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 11:58:00.299111] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 11:58:00.299163] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 11:58:00.299236] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:54:34.676035 (xid=0x2503x) >[2012-10-04 11:58:00.300247] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 11:58:00.300300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:01:27.329127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:01:27.329182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:01:27.329250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 11:58:01.706098 (xid=0x2504x) >[2012-10-04 12:01:27.329271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:01:27.329287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:02:09.746763] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:02:09.750276] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:02:09.753367] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:04:54.360128] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:04:54.360180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:04:54.360247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:01:28.736967 (xid=0x2505x) >[2012-10-04 12:04:54.360269] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:04:54.360286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:08:21.393075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:08:21.393126] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:08:21.393194] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:04:55.769923 (xid=0x2506x) >[2012-10-04 12:08:21.393216] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:08:21.393232] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:11:48.422155] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:11:48.422213] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:11:48.422304] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:08:22.799270 (xid=0x2507x) >[2012-10-04 12:11:48.422325] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:11:48.422342] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:12:09.835923] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:12:09.838673] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:12:09.840933] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:15:15.452109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:15:15.452161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:15:15.452228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:11:49.829052 (xid=0x2508x) >[2012-10-04 12:15:15.452249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:15:15.452266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:18:42.485103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:18:42.485160] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:18:42.485228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:15:16.862487 (xid=0x2509x) >[2012-10-04 12:18:42.485249] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:18:42.485266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:22:09.516125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:22:09.516177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:22:09.516243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:18:43.893433 (xid=0x2510x) >[2012-10-04 12:22:09.516276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:22:09.516294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:22:09.923111] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:22:09.926251] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:22:09.928924] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:25:36.549130] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:25:36.549182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:25:36.549250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:22:10.925918 (xid=0x2511x) >[2012-10-04 12:25:36.549272] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:25:36.549288] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:29:03.582107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:29:03.582161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:29:03.582228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:25:37.959397 (xid=0x2512x) >[2012-10-04 12:29:03.582250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:29:03.582267] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:32:10.016763] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:32:10.020086] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:32:10.022927] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:32:30.612135] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:32:30.612181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:32:30.612246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:29:04.989034 (xid=0x2513x) >[2012-10-04 12:32:30.612266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:32:30.612281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:35:57.643127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:35:57.643179] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:35:57.643246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:32:32.020173 (xid=0x2514x) >[2012-10-04 12:35:57.643268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:35:57.643284] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:39:24.675107] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:39:24.675161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:39:24.675229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:35:59.052471 (xid=0x2515x) >[2012-10-04 12:39:24.675251] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:39:24.675268] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:42:10.111189] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:5d255c3d-7d8a-4458-98e4-0762e05bf0a8>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:42:10.114455] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:c2e13dc5-572d-4ee1-8bb9-ce0e32177a74>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:42:10.117352] E [afr-self-heal-data.c:763:afr_sh_data_fxattrop_fstat_done] 0-replicate-rhevh2-replicate-0: Unable to self-heal contents of '<gfid:2844ea62-845e-42ce-b33f-dcb93edb4700>' (possible split-brain). Please delete the file from all but the preferred subvolume. >[2012-10-04 12:42:51.706129] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:42:51.706181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:42:51.706257] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:39:26.083690 (xid=0x2516x) >[2012-10-04 12:42:51.706278] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:42:51.706293] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:46:18.741118] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:46:18.741168] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:46:18.741234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:42:53.118069 (xid=0x2517x) >[2012-10-04 12:46:18.741256] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:46:18.741271] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:49:45.774094] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:49:45.774164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:49:45.774246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:46:20.151165 (xid=0x2518x) >[2012-10-04 12:49:45.774268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:49:45.774286] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:53:12.805132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:53:12.805180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:53:12.805247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:49:47.182013 (xid=0x2519x) >[2012-10-04 12:53:12.805268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:53:12.805283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 12:55:25.701830] W [socket.c:1512:__socket_proto_state_machine] 0-replicate-rhevh2-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.36.30:24013) >[2012-10-04 12:55:25.701895] I [client.c:2090:client_rpc_notify] 0-replicate-rhevh2-client-0: disconnected >[2012-10-04 12:55:36.236464] E [socket.c:1715:socket_connect_finish] 0-replicate-rhevh2-client-0: connection to 10.70.36.30:24013 failed (Connection refused) >[2012-10-04 12:56:39.837100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 12:56:39.837165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 12:56:39.837241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:53:14.213968 (xid=0x2520x) >[2012-10-04 12:56:39.838264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 12:56:39.838309] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:00:06.961127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:00:06.961177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:00:06.961244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 12:56:41.338027 (xid=0x2521x) >[2012-10-04 13:00:06.961266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:00:06.961283] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:02:10.830879] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 13:03:34.270089] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:03:34.270140] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:03:34.270206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:00:08.647817 (xid=0x2522x) >[2012-10-04 13:03:34.270227] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:03:34.270243] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:07:01.957068] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:07:01.957131] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:07:01.957208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:03:36.334733 (xid=0x2523x) >[2012-10-04 13:07:01.957230] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:07:01.957248] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:07:08.665053] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 13:07:08.668084] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:07:08.668577] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:07:08.669145] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:07:08.669646] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:07:08.670111] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:07:08.670316] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:07:08.670737] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:07:08.671002] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:08:07.846250] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 13:08:07.850032] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:08:07.850535] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:08:07.850972] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:08:07.851419] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:08:07.851886] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:08:07.852095] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:08:07.852536] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:08:07.852762] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:10:29.271103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:10:29.271170] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:10:29.271254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:07:03.648201 (xid=0x2524x) >[2012-10-04 13:10:29.271289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:10:29.271307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:11:46.240676] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 13:11:46.245554] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:11:46.245942] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:11:46.246420] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:11:46.246790] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:11:46.247195] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:11:46.247380] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:11:46.247846] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:11:46.248181] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:12:11.111372] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 13:13:56.584028] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:13:56.584103] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:13:56.584186] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:10:30.961327 (xid=0x2525x) >[2012-10-04 13:13:56.584209] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:13:56.584226] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:17:23.903037] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:17:23.903111] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:17:23.903192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:13:58.280146 (xid=0x2526x) >[2012-10-04 13:17:23.903214] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:17:23.903231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:20:51.220052] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:20:51.220455] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:20:51.220551] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:17:25.597390 (xid=0x2527x) >[2012-10-04 13:20:51.220575] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:20:51.220593] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:21:20.268889] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 13:21:20.272780] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:21:20.273164] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:21:20.274140] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:21:20.274581] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:21:20.275159] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:21:20.275393] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:21:20.276005] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:21:20.276249] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:22:12.036072] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 13:24:18.539063] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:24:18.539138] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:24:18.539220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:20:52.916100 (xid=0x2528x) >[2012-10-04 13:24:18.539243] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:24:18.539260] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:27:45.889106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:27:45.889157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:27:45.889238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:24:20.266750 (xid=0x2529x) >[2012-10-04 13:27:45.889259] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:27:45.889275] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:28:45.220585] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 13:28:45.225224] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:28:45.225835] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:28:45.226513] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:28:45.227107] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:28:45.227757] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:28:45.228042] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:28:45.228626] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:28:45.228837] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:31:13.233112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:31:13.233161] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:31:13.233227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:27:47.610287 (xid=0x2530x) >[2012-10-04 13:31:13.233248] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:31:13.233264] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:32:12.046671] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 13:34:40.564126] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:34:40.564184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:34:40.564256] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:31:14.941066 (xid=0x2531x) >[2012-10-04 13:34:40.564293] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:34:40.564311] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:35:45.317839] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 13:35:45.322313] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:35:45.323000] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:35:45.323618] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:35:45.324181] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:35:45.324636] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:35:45.324867] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:35:45.325359] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:35:45.325572] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:38:07.908106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:38:07.908157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:38:07.908223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:34:42.285310 (xid=0x2532x) >[2012-10-04 13:38:07.908244] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:38:07.908261] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:41:35.248132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:41:35.248182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:41:35.248247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:38:09.625724 (xid=0x2533x) >[2012-10-04 13:41:35.248268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:41:35.248296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:42:13.015361] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 13:45:02.569106] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:45:02.569154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:45:02.569219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:41:36.946634 (xid=0x2534x) >[2012-10-04 13:45:02.569240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:45:02.569255] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:48:29.929137] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:48:29.929185] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:48:29.929250] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:45:04.306010 (xid=0x2535x) >[2012-10-04 13:48:29.929271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:48:29.929287] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:48:44.634652] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 13:48:44.638871] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:48:44.639440] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:48:44.639887] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:48:44.640442] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:48:44.640954] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:48:44.641196] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:48:44.641670] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 13:48:44.641854] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 13:51:57.572116] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:51:57.572178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:51:57.572244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:48:31.949285 (xid=0x2536x) >[2012-10-04 13:51:57.572264] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:51:57.572280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:52:13.283328] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 13:55:27.882108] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:55:27.882154] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:55:27.882219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:52:02.259158 (xid=0x2537x) >[2012-10-04 13:55:27.882240] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:55:27.882256] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 13:58:55.200112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 13:58:55.200156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 13:58:55.200220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:55:29.577682 (xid=0x2538x) >[2012-10-04 13:58:55.200242] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 13:58:55.200258] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:02:14.185604] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 14:02:22.512098] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:02:22.512141] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:02:22.512205] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 13:58:56.889249 (xid=0x2539x) >[2012-10-04 14:02:22.512226] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:02:22.512241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:05:49.827088] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:05:49.827147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:05:49.827212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:02:24.204573 (xid=0x2540x) >[2012-10-04 14:05:49.827233] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:05:49.827249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:09:17.140122] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:09:17.140167] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:09:17.140232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:05:51.517022 (xid=0x2541x) >[2012-10-04 14:09:17.140253] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:09:17.140270] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:11:40.099547] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 14:11:40.103632] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:11:40.104237] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:11:40.104772] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:11:40.105217] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:11:40.105679] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 14:11:40.105883] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:11:40.106462] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 14:11:40.106654] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:12:15.089914] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 14:12:44.449142] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:12:44.449188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:12:44.449267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:09:18.826720 (xid=0x2542x) >[2012-10-04 14:12:44.449288] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:12:44.449305] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:16:11.763091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:16:11.763143] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:16:11.763210] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:12:46.140633 (xid=0x2543x) >[2012-10-04 14:16:11.763232] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:16:11.763249] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:19:39.081103] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:19:39.081164] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:19:39.081237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:16:13.458173 (xid=0x2544x) >[2012-10-04 14:19:39.081262] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:19:39.081280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:22:16.004474] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 14:23:06.395138] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:23:06.395194] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:23:06.395262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:19:40.772361 (xid=0x2545x) >[2012-10-04 14:23:06.395285] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:23:06.395302] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:26:33.713104] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:26:33.713158] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:26:33.713225] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:23:08.090371 (xid=0x2546x) >[2012-10-04 14:26:33.713263] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:26:33.713280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:30:01.024066] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:30:01.024121] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:30:01.024189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:26:35.401563 (xid=0x2547x) >[2012-10-04 14:30:01.024213] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:30:01.024231] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:31:48.495889] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 14:31:48.500476] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:31:48.501091] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:31:48.501746] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:31:48.502423] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:31:48.503069] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 14:31:48.503475] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:31:48.504135] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 14:31:48.504427] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:32:16.912609] E [afr-self-heald.c:418:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl as < 2 children are up >[2012-10-04 14:33:28.337080] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:33:28.337126] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:33:28.337189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:30:02.714595 (xid=0x2548x) >[2012-10-04 14:33:28.337223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:33:28.337240] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:36:55.648079] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:36:55.648125] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:36:55.648188] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:33:30.025093 (xid=0x2549x) >[2012-10-04 14:36:55.648210] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:36:55.648226] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:40:22.961132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:40:22.961178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:40:22.961243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:36:57.338741 (xid=0x2550x) >[2012-10-04 14:40:22.961265] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:40:22.961281] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:41:37.293801] E [afr-self-heald.c:409:_crawl_proceed] 0-replicate-rhevh2-replicate-0: Stopping crawl for replicate-rhevh2-client-0 , subvol went down >[2012-10-04 14:41:37.298908] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:37.299558] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:37.300234] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:37.300897] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:37.301695] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:00ccc0bf-9682-4532-a4aa-3a23eb0e6b56> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 14:41:37.302043] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:37.302657] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-replicate-rhevh2-client-1: remote operation failed: No such file or directory. Path: <gfid:93d43247-6405-49b8-9fca-d59cb7911743> (00000000-0000-0000-0000-000000000000). Key: glusterfs.gfid2path >[2012-10-04 14:41:37.302960] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:44.482658] W [glusterfsd.c:906:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x322d2e5ccd] (-->/lib64/libpthread.so.0() [0x322da077f1] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d2d]))) 0-: received signum (15), shutting down >[2012-10-04 14:41:45.488669] I [glusterfsd.c:1741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0rhsvirt1 >[2012-10-04 14:41:45.534656] I [graph.c:241:gf_add_cmdline_options] 0-replicate-rhevh2-replicate-0: adding option 'node-uuid' for volume 'replicate-rhevh2-replicate-0' with value 'b9d6cb21-051f-4791-9476-734856e77fbf' >[2012-10-04 14:41:45.534715] I [graph.c:241:gf_add_cmdline_options] 0-replicate-rhevh2-client-1: adding option 'node-uuid' for volume 'replicate-rhevh2-client-1' with value 'b9d6cb21-051f-4791-9476-734856e77fbf' >[2012-10-04 14:41:45.534737] I [graph.c:241:gf_add_cmdline_options] 0-replicate-rhevh2-client-0: adding option 'node-uuid' for volume 'replicate-rhevh2-client-0' with value 'b9d6cb21-051f-4791-9476-734856e77fbf' >[2012-10-04 14:41:45.534753] I [graph.c:241:gf_add_cmdline_options] 0-dist-rep-rhevh-replicate-1: adding option 'node-uuid' for volume 'dist-rep-rhevh-replicate-1' with value 'b9d6cb21-051f-4791-9476-734856e77fbf' >[2012-10-04 14:41:45.534779] I [graph.c:241:gf_add_cmdline_options] 0-dist-rep-rhevh-replicate-0: adding option 'node-uuid' for volume 'dist-rep-rhevh-replicate-0' with value 'b9d6cb21-051f-4791-9476-734856e77fbf' >[2012-10-04 14:41:45.547096] W [graph.c:316:_log_if_unknown_option] 0-replicate-rhevh2-client-1: option 'node-uuid' is not recognized >[2012-10-04 14:41:45.547181] W [graph.c:316:_log_if_unknown_option] 0-replicate-rhevh2-client-0: option 'node-uuid' is not recognized >[2012-10-04 14:41:45.547222] I [client.c:2142:notify] 0-replicate-rhevh2-client-0: parent translators are ready, attempting connect on transport >[2012-10-04 14:41:45.551987] I [client.c:2142:notify] 0-replicate-rhevh2-client-1: parent translators are ready, attempting connect on transport >[2012-10-04 14:41:45.556211] I [client.c:2142:notify] 0-dist-rep-rhevh-client-0: parent translators are ready, attempting connect on transport >[2012-10-04 14:41:45.561081] I [client.c:2142:notify] 0-dist-rep-rhevh-client-1: parent translators are ready, attempting connect on transport >[2012-10-04 14:41:45.565464] I [client.c:2142:notify] 0-dist-rep-rhevh-client-2: parent translators are ready, attempting connect on transport >[2012-10-04 14:41:45.569572] I [client.c:2142:notify] 0-dist-rep-rhevh-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume dist-rep-rhevh-client-0 > 2: type protocol/client > 3: option remote-host rhs-client6.lab.eng.blr.redhat.com > 4: option remote-subvolume /disk1 > 5: option transport-type tcp > 6: option username a6c017a0-c3d4-4411-aee5-fc1e0c88e5a8 > 7: option password 1233f788-e862-447a-9353-3a50d84656ca > 8: end-volume > 9: > 10: volume dist-rep-rhevh-client-1 > 11: type protocol/client > 12: option remote-host rhs-client7.lab.eng.blr.redhat.com > 13: option remote-subvolume /disk1 > 14: option transport-type tcp > 15: option username a6c017a0-c3d4-4411-aee5-fc1e0c88e5a8 > 16: option password 1233f788-e862-447a-9353-3a50d84656ca > 17: end-volume > 18: > 19: volume dist-rep-rhevh-client-2 > 20: type protocol/client > 21: option remote-host rhs-client8.lab.eng.blr.redhat.com > 22: option remote-subvolume /disk1 > 23: option transport-type tcp > 24: option username a6c017a0-c3d4-4411-aee5-fc1e0c88e5a8 > 25: option password 1233f788-e862-447a-9353-3a50d84656ca > 26: end-volume > 27: > 28: volume dist-rep-rhevh-client-3 > 29: type protocol/client > 30: option remote-host rhs-client9.lab.eng.blr.redhat.com > 31: option remote-subvolume /disk1 > 32: option transport-type tcp > 33: option username a6c017a0-c3d4-4411-aee5-fc1e0c88e5a8 > 34: option password 1233f788-e862-447a-9353-3a50d84656ca > 35: end-volume > 36: > 37: volume dist-rep-rhevh-replicate-0 > 38: type cluster/replicate > 39: option background-self-heal-count 0 > 40: option metadata-self-heal on > 41: option data-self-heal on > 42: option entry-self-heal on > 43: option self-heal-daemon on > 44: option eager-lock enable > 45: option iam-self-heal-daemon yes > 46: subvolumes dist-rep-rhevh-client-0 dist-rep-rhevh-client-1 > 47: end-volume > 48: > 49: volume dist-rep-rhevh-replicate-1 > 50: type cluster/replicate > 51: option background-self-heal-count 0 > 52: option metadata-self-heal on > 53: option data-self-heal on > 54: option entry-self-heal on > 55: option self-heal-daemon on > 56: option eager-lock enable > 57: option iam-self-heal-daemon yes > 58: subvolumes dist-rep-rhevh-client-2 dist-rep-rhevh-client-3 > 59: end-volume > 60: > 61: volume replicate-rhevh2-client-0 > 62: type protocol/client > 63: option remote-host rhs-client6.lab.eng.blr.redhat.com > 64: option remote-subvolume /replicate-disk > 65: option transport-type tcp > 66: option username ed47646f-f966-4cda-9e0f-1417c35dd7c1 > 67: option password 3c031854-e508-409c-a58d-0fc991070ed0 > 68: end-volume > 69: > 70: volume replicate-rhevh2-client-1 > 71: type protocol/client > 72: option remote-host rhs-client7.lab.eng.blr.redhat.com > 73: option remote-subvolume /replicate-disk > 74: option transport-type tcp > 75: option username ed47646f-f966-4cda-9e0f-1417c35dd7c1 > 76: option password 3c031854-e508-409c-a58d-0fc991070ed0 > 77: end-volume > 78: > 79: volume replicate-rhevh2-replicate-0 > 80: type cluster/replicate > 81: option background-self-heal-count 0 > 82: option metadata-self-heal on > 83: option data-self-heal on > 84: option entry-self-heal on > 85: option self-heal-daemon on > 86: option eager-lock enable > 87: option iam-self-heal-daemon yes > 88: subvolumes replicate-rhevh2-client-0 replicate-rhevh2-client-1 > 89: end-volume > 90: > 91: volume glustershd > 92: type debug/io-stats > 93: subvolumes replicate-rhevh2-replicate-0 dist-rep-rhevh-replicate-0 dist-rep-rhevh-replicate-1 > 94: end-volume > >+------------------------------------------------------------------------------+ >[2012-10-04 14:41:45.574900] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-replicate-rhevh2-client-1: changing port to 24013 (from 0) >[2012-10-04 14:41:45.575023] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-dist-rep-rhevh-client-1: changing port to 24011 (from 0) >[2012-10-04 14:41:45.575096] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-replicate-rhevh2-client-0: changing port to 24014 (from 0) >[2012-10-04 14:41:45.575160] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-dist-rep-rhevh-client-0: changing port to 24011 (from 0) >[2012-10-04 14:41:46.080140] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-dist-rep-rhevh-client-2: changing port to 24011 (from 0) >[2012-10-04 14:41:48.615459] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-10-04 14:41:48.615509] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-10-04 14:41:49.504191] I [client-handshake.c:1614:select_server_supported_programs] 0-replicate-rhevh2-client-1: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330) >[2012-10-04 14:41:49.504478] I [client-handshake.c:1411:client_setvolume_cbk] 0-replicate-rhevh2-client-1: Connected to 10.70.36.31:24013, attached to remote volume '/replicate-disk'. >[2012-10-04 14:41:49.504530] I [client-handshake.c:1423:client_setvolume_cbk] 0-replicate-rhevh2-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-10-04 14:41:49.504629] I [afr-common.c:3631:afr_notify] 0-replicate-rhevh2-replicate-0: Subvolume 'replicate-rhevh2-client-1' came back up; going online. >[2012-10-04 14:41:49.504806] I [client-handshake.c:453:client_set_lk_version_cbk] 0-replicate-rhevh2-client-1: Server lk version = 1 >[2012-10-04 14:41:49.506492] E [afr-self-heal-data.c:1311:afr_sh_data_open_cbk] 0-replicate-rhevh2-replicate-0: open of <gfid:c874c724-0853-4f2d-a6e3-087549630ec7> failed on child replicate-rhevh2-client-0 (Transport endpoint is not connected) >[2012-10-04 14:41:49.521848] I [client-handshake.c:1614:select_server_supported_programs] 0-dist-rep-rhevh-client-1: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330) >[2012-10-04 14:41:49.521954] E [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk] 0-replicate-rhevh2-replicate-0: Non Blocking entrylks failed for <gfid:a337070f-5327-43b0-bedd-fdfcbdaddba9>. >[2012-10-04 14:41:49.522053] I [client-handshake.c:1614:select_server_supported_programs] 0-replicate-rhevh2-client-0: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330) >[2012-10-04 14:41:49.522149] I [client-handshake.c:1614:select_server_supported_programs] 0-dist-rep-rhevh-client-0: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330) >[2012-10-04 14:41:49.522229] I [client-handshake.c:1411:client_setvolume_cbk] 0-dist-rep-rhevh-client-1: Connected to 10.70.36.31:24011, attached to remote volume '/disk1'. >[2012-10-04 14:41:49.522252] I [client-handshake.c:1423:client_setvolume_cbk] 0-dist-rep-rhevh-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-10-04 14:41:49.522314] I [afr-common.c:3631:afr_notify] 0-dist-rep-rhevh-replicate-0: Subvolume 'dist-rep-rhevh-client-1' came back up; going online. >[2012-10-04 14:41:49.522456] I [client-handshake.c:453:client_set_lk_version_cbk] 0-dist-rep-rhevh-client-1: Server lk version = 1 >[2012-10-04 14:41:49.522510] I [client-handshake.c:1411:client_setvolume_cbk] 0-replicate-rhevh2-client-0: Connected to 10.70.36.30:24014, attached to remote volume '/replicate-disk'. >[2012-10-04 14:41:49.522532] I [client-handshake.c:1423:client_setvolume_cbk] 0-replicate-rhevh2-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-10-04 14:41:49.522745] I [client-handshake.c:1411:client_setvolume_cbk] 0-dist-rep-rhevh-client-0: Connected to 10.70.36.30:24011, attached to remote volume '/disk1'. >[2012-10-04 14:41:49.522772] I [client-handshake.c:1423:client_setvolume_cbk] 0-dist-rep-rhevh-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-10-04 14:41:49.523008] I [client-handshake.c:453:client_set_lk_version_cbk] 0-replicate-rhevh2-client-0: Server lk version = 1 >[2012-10-04 14:41:49.523068] I [client-handshake.c:453:client_set_lk_version_cbk] 0-dist-rep-rhevh-client-0: Server lk version = 1 >[2012-10-04 14:41:49.523657] W [client3_1-fops.c:1550:client3_1_inodelk_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory >[2012-10-04 14:41:49.524250] W [client3_1-fops.c:1650:client3_1_entrylk_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory >[2012-10-04 14:41:49.524477] E [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk] 0-replicate-rhevh2-replicate-0: Non Blocking entrylks failed for <gfid:39aa0c8a-b87f-4441-8f6f-b40e54be7a5f>. >[2012-10-04 14:41:49.525554] W [client3_1-fops.c:1550:client3_1_inodelk_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory >[2012-10-04 14:41:49.525850] I [client-handshake.c:1614:select_server_supported_programs] 0-dist-rep-rhevh-client-2: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330) >[2012-10-04 14:41:49.526152] W [client3_1-fops.c:473:client3_1_open_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory. Path: <gfid:ca655693-09fa-4769-8742-e29d884ceca9> (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:49.526184] E [afr-self-heal-data.c:1311:afr_sh_data_open_cbk] 0-replicate-rhevh2-replicate-0: open of <gfid:ca655693-09fa-4769-8742-e29d884ceca9> failed on child replicate-rhevh2-client-0 (No such file or directory) >[2012-10-04 14:41:49.526305] I [client-handshake.c:1411:client_setvolume_cbk] 0-dist-rep-rhevh-client-2: Connected to 10.70.36.32:24011, attached to remote volume '/disk1'. >[2012-10-04 14:41:49.526343] I [client-handshake.c:1423:client_setvolume_cbk] 0-dist-rep-rhevh-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-10-04 14:41:49.526439] I [afr-common.c:3631:afr_notify] 0-dist-rep-rhevh-replicate-1: Subvolume 'dist-rep-rhevh-client-2' came back up; going online. >[2012-10-04 14:41:49.526634] I [client-handshake.c:453:client_set_lk_version_cbk] 0-dist-rep-rhevh-client-2: Server lk version = 1 >[2012-10-04 14:41:49.528230] W [client3_1-fops.c:1550:client3_1_inodelk_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory >[2012-10-04 14:41:49.528872] W [client3_1-fops.c:473:client3_1_open_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory. Path: <gfid:d4d6cf31-1a8f-44f1-b707-12ad56464a83> (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:49.528902] E [afr-self-heal-data.c:1311:afr_sh_data_open_cbk] 0-replicate-rhevh2-replicate-0: open of <gfid:d4d6cf31-1a8f-44f1-b707-12ad56464a83> failed on child replicate-rhevh2-client-0 (No such file or directory) >[2012-10-04 14:41:49.530668] W [client3_1-fops.c:1550:client3_1_inodelk_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory >[2012-10-04 14:41:49.531203] W [client3_1-fops.c:473:client3_1_open_cbk] 0-replicate-rhevh2-client-0: remote operation failed: No such file or directory. Path: <gfid:f16a4d71-3133-4d9c-8f3f-36dd64ed8feb> (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:41:49.531252] E [afr-self-heal-data.c:1311:afr_sh_data_open_cbk] 0-replicate-rhevh2-replicate-0: open of <gfid:f16a4d71-3133-4d9c-8f3f-36dd64ed8feb> failed on child replicate-rhevh2-client-0 (No such file or directory) >[2012-10-04 14:42:05.544801] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:42:05.545594] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:42:05.546423] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:42:05.547256] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:42:05.547964] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:42:05.548685] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:42:47.276313] W [dict.c:2339:dict_unserialize] (-->/lib64/libc.so.6() [0x322d243610] (-->/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12) [0x3a9f24bd72] (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) [0x40907f]))) 0-dict: buf is null! >[2012-10-04 14:42:47.276397] E [glusterfsd-mgmt.c:672:glusterfs_handle_translator_op] 0-glusterfs: failed to unserialize req-buffer to dictionary >[2012-10-04 14:43:09.690339] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:43:09.690968] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:43:09.691490] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:43:09.692166] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:43:48.655951] W [dict.c:2339:dict_unserialize] (-->/lib64/libc.so.6() [0x322d243610] (-->/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12) [0x3a9f24bd72] (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) [0x40907f]))) 0-dict: buf is null! >[2012-10-04 14:43:48.656032] E [glusterfsd-mgmt.c:672:glusterfs_handle_translator_op] 0-glusterfs: failed to unserialize req-buffer to dictionary >[2012-10-04 14:44:04.242844] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:04.243393] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:04.244017] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:04.244617] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:27.609925] W [dict.c:2339:dict_unserialize] (-->/lib64/libc.so.6() [0x322d243610] (-->/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12) [0x3a9f24bd72] (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) [0x40907f]))) 0-dict: buf is null! >[2012-10-04 14:44:27.610029] E [glusterfsd-mgmt.c:672:glusterfs_handle_translator_op] 0-glusterfs: failed to unserialize req-buffer to dictionary >[2012-10-04 14:44:34.735227] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:34.735933] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:34.736447] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:34.736981] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:44:39.649150] W [dict.c:2339:dict_unserialize] (-->/lib64/libc.so.6() [0x322d243610] (-->/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12) [0x3a9f24bd72] (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) [0x40907f]))) 0-dict: buf is null! >[2012-10-04 14:44:39.649238] E [glusterfsd-mgmt.c:672:glusterfs_handle_translator_op] 0-glusterfs: failed to unserialize req-buffer to dictionary >[2012-10-04 14:45:14.197075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:45:14.197172] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:45:14.197262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:41:48.574146 (xid=0x1x) >[2012-10-04 14:45:14.197289] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:45:14.197309] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:45:51.792662] W [dict.c:2339:dict_unserialize] (-->/lib64/libc.so.6() [0x322d243610] (-->/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12) [0x3a9f24bd72] (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) [0x40907f]))) 0-dict: buf is null! >[2012-10-04 14:45:51.792734] E [glusterfsd-mgmt.c:672:glusterfs_handle_translator_op] 0-glusterfs: failed to unserialize req-buffer to dictionary >[2012-10-04 14:45:55.487879] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:45:55.488845] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:45:55.489494] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:45:55.490094] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:48:42.177176] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:48:42.177252] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:48:42.177326] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:45:16.554048 (xid=0x2x) >[2012-10-04 14:48:42.177352] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:48:42.177371] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:49:32.147732] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:49:32.148321] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:51:38.935727] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:51:38.936334] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:52:09.205991] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:52:09.206086] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:52:09.206166] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:48:43.583588 (xid=0x3x) >[2012-10-04 14:52:09.206200] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:52:09.206219] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:52:25.017184] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:52:25.017937] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:54:39.997472] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:54:40.008692] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:55:36.234005] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:55:36.234092] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:55:36.234173] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:52:10.611770 (xid=0x4x) >[2012-10-04 14:55:36.234206] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:55:36.234224] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:56:13.383129] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:56:13.385531] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:59:03.258167] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 14:59:03.258238] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 14:59:03.258316] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:55:37.635745 (xid=0x5x) >[2012-10-04 14:59:03.258341] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 14:59:03.258361] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 14:59:53.566229] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 14:59:53.567025] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:02:30.286996] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:02:30.287087] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:02:30.287166] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 14:59:04.665057 (xid=0x6x) >[2012-10-04 15:02:30.287191] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:02:30.287209] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:05:57.321014] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:05:57.321106] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:05:57.321189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:02:31.698818 (xid=0x7x) >[2012-10-04 15:05:57.321214] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:05:57.321251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:06:38.392532] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:06:38.393178] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:09:24.346109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:09:24.346180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:09:24.346273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:05:58.723015 (xid=0x8x) >[2012-10-04 15:09:24.346300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:09:24.346330] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:12:48.831244] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:12:48.832289] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:12:51.377133] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:12:51.377247] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:12:51.377352] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:09:25.754313 (xid=0x9x) >[2012-10-04 15:12:51.377390] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:12:51.377412] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:16:18.407022] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:16:18.407104] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:16:18.407194] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:12:52.783232 (xid=0x10x) >[2012-10-04 15:16:18.407223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:16:18.407251] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:19:45.430081] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:19:45.430157] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:19:45.430255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:16:19.807243 (xid=0x11x) >[2012-10-04 15:19:45.430300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:19:45.430330] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:21:49.844337] I [afr-self-heald.c:1082:afr_dir_exclusive_crawl] 0-replicate-rhevh2-replicate-0: Another crawl is in progress for replicate-rhevh2-client-1 >[2012-10-04 15:23:12.454064] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:23:12.454146] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:23:12.454237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:19:46.830921 (xid=0x12x) >[2012-10-04 15:23:12.454266] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:23:12.454294] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:26:39.480978] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:26:39.481079] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:26:39.481174] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:23:13.857175 (xid=0x13x) >[2012-10-04 15:26:39.481202] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:26:39.481232] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:30:06.502100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:30:06.502209] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:30:06.502310] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:26:40.879194 (xid=0x14x) >[2012-10-04 15:30:06.502348] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:30:06.502366] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:31:49.915824] I [afr-self-heald.c:1082:afr_dir_exclusive_crawl] 0-replicate-rhevh2-replicate-0: Another crawl is in progress for replicate-rhevh2-client-1 >[2012-10-04 15:33:33.526048] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:33:33.526127] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:33:33.526240] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:30:07.903164 (xid=0x15x) >[2012-10-04 15:33:33.526268] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:33:33.526296] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:37:00.555099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:37:00.555180] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:37:00.555274] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:33:34.932231 (xid=0x16x) >[2012-10-04 15:37:00.555304] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:37:00.555334] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:40:30.578093] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:40:30.578177] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:40:30.578278] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:37:04.955125 (xid=0x17x) >[2012-10-04 15:40:30.578305] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:40:30.578338] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:41:43.516103] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:41:43.518393] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:41:49.989226] I [afr-self-heald.c:1082:afr_dir_exclusive_crawl] 0-replicate-rhevh2-replicate-0: Another crawl is in progress for replicate-rhevh2-client-1 >[2012-10-04 15:43:57.604015] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:43:57.604117] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:43:57.604219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:40:31.981522 (xid=0x18x) >[2012-10-04 15:43:57.604246] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:43:57.604277] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:45:17.044495] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:45:17.045479] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 15:47:24.626990] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:47:24.627094] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:47:24.627192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:43:59.004669 (xid=0x19x) >[2012-10-04 15:47:24.627218] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:47:24.627238] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:50:51.650092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:50:51.650156] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:50:51.650249] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:47:26.027789 (xid=0x20x) >[2012-10-04 15:50:51.650276] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:50:51.650307] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:54:18.678030] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:54:18.678127] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:54:18.678223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:50:53.055608 (xid=0x21x) >[2012-10-04 15:54:18.678250] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:54:18.678280] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 15:57:45.705109] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 15:57:45.705178] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 15:57:45.705270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:54:20.082357 (xid=0x22x) >[2012-10-04 15:57:45.705297] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 15:57:45.705343] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:01:12.733148] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:01:12.733211] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:01:12.733302] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 15:57:47.110542 (xid=0x23x) >[2012-10-04 16:01:12.733329] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:01:12.733362] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:04:39.763083] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:04:39.763165] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:04:39.763260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:01:14.140409 (xid=0x24x) >[2012-10-04 16:04:39.763287] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:04:39.763317] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:08:06.789136] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:08:06.789198] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:08:06.789299] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:04:41.166219 (xid=0x25x) >[2012-10-04 16:08:06.789327] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:08:06.789353] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:11:33.818112] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:11:33.818176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:11:33.818264] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:08:08.195736 (xid=0x26x) >[2012-10-04 16:11:33.818292] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:11:33.818322] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:15:00.852092] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:15:00.852184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:15:00.852286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:11:35.228914 (xid=0x27x) >[2012-10-04 16:15:00.852314] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:15:00.852344] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:18:27.884127] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:18:27.884193] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:18:27.884286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:15:02.261255 (xid=0x28x) >[2012-10-04 16:18:27.884314] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:18:27.884346] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:21:57.915959] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:21:57.916039] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:21:57.916147] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:18:32.293127 (xid=0x29x) >[2012-10-04 16:21:57.916175] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:21:57.916211] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:25:24.952073] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:25:24.952147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:25:24.952243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:21:59.328851 (xid=0x30x) >[2012-10-04 16:25:24.952271] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:25:24.952300] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:28:51.976100] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:28:51.976169] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:28:51.976283] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:25:26.353620 (xid=0x31x) >[2012-10-04 16:28:51.976310] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:28:51.976348] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:31:30.335918] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 16:31:30.336737] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 16:32:19.004014] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:32:19.004099] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:32:19.004197] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:28:53.381823 (xid=0x32x) >[2012-10-04 16:32:19.004223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:32:19.004252] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:35:46.034995] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now >[2012-10-04 16:35:46.035161] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:32:20.412032 (xid=0x33x) >[2012-10-04 16:35:46.035187] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:35:46.035206] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:39:13.058114] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:39:13.058181] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:39:13.058273] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:35:47.435137 (xid=0x34x) >[2012-10-04 16:39:13.058300] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:39:13.058330] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:42:40.087001] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:42:40.087100] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:42:40.087185] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:39:14.464278 (xid=0x35x) >[2012-10-04 16:42:40.087223] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:42:40.087241] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:46:10.117091] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:46:10.117174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:46:10.117275] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:42:44.494132 (xid=0x36x) >[2012-10-04 16:46:10.117302] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:46:10.117331] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:49:37.144125] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:49:37.144188] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:49:37.144280] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:46:11.520836 (xid=0x37x) >[2012-10-04 16:49:37.144307] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:49:37.144328] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:53:04.172099] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:53:04.172230] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:53:04.172342] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:49:38.549027 (xid=0x38x) >[2012-10-04 16:53:04.172369] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:53:04.172394] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:53:04.769326] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 16:53:04.770363] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 16:56:00.339550] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 16:56:00.340304] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 16:56:31.199042] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:56:31.199122] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:56:31.199212] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:53:05.576416 (xid=0x39x) >[2012-10-04 16:56:31.199238] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:56:31.199266] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 16:59:58.223113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 16:59:58.223182] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 16:59:58.223284] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:56:32.600596 (xid=0x40x) >[2012-10-04 16:59:58.223991] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 16:59:58.224163] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:03:25.250075] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:03:25.250147] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:03:25.250253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 16:59:59.627773 (xid=0x41x) >[2012-10-04 17:03:25.250280] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:03:25.250312] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:06:52.285113] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:06:52.285176] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:06:52.285269] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:03:26.662573 (xid=0x42x) >[2012-10-04 17:06:52.285296] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:06:52.285331] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:06:54.561873] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 17:06:54.562561] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 17:10:19.312132] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:10:19.312322] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:10:19.312414] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:06:53.689871 (xid=0x43x) >[2012-10-04 17:10:19.312442] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:10:19.312472] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:11:10.692353] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 17:11:10.692899] E [afr-self-heald.c:685:_link_inode_update_loc] 0-replicate-rhevh2-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000) >[2012-10-04 17:13:46.345987] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:13:46.346069] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:13:46.346157] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:10:20.723871 (xid=0x44x) >[2012-10-04 17:13:46.346182] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:13:46.346201] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:17:13.375105] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:17:13.375174] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:17:13.375265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:13:47.752542 (xid=0x45x) >[2012-10-04 17:17:13.375291] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:17:13.375327] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:20:40.405139] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:20:40.405200] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:20:40.405292] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:17:14.782099 (xid=0x46x) >[2012-10-04 17:20:40.405333] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:20:40.405365] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:24:07.435071] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:24:07.435462] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:24:07.435556] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:20:41.812852 (xid=0x47x) >[2012-10-04 17:24:07.435581] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:24:07.435607] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:27:34.464170] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:27:34.464236] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:27:34.464327] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:24:08.841196 (xid=0x48x) >[2012-10-04 17:27:34.464354] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:27:34.464383] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected >[2012-10-04 17:31:01.489121] W [socket.c:195:__socket_rwv] 0-dist-rep-rhevh-client-3: readv failed (Connection reset by peer) >[2012-10-04 17:31:01.489184] W [socket.c:1512:__socket_proto_state_machine] 0-dist-rep-rhevh-client-3: reading from socket failed. Error (Connection reset by peer), peer (10.70.36.33:24007) >[2012-10-04 17:31:01.489277] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-04 17:27:35.866367 (xid=0x49x) >[2012-10-04 17:31:01.489304] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error >[2012-10-04 17:31:01.489336] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 863084
: 621592