Attachment 615411 Details for Bug 859387 – rebalance-migration failure logs

rebalance-migration failure logs

rebalance-failures.log (text/x-log), 118.46 KB, created by shylesh on 2012-09-21 12:12:25 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: shylesh

Created: 2012-09-21 12:12:25 UTC

Size: 118.46 KB

patch

obsolete

>[2012-09-21 06:57:08.360444] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-09-21 06:57:08.360533] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-09-21 06:57:08.363140] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-dht: adding option 'node-uuid' for volume 'distrep2-dht' with value '9b9e136b-1d14-421c-ad6a-2ab6851562c7'
>[2012-09-21 06:57:08.363164] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-dht: adding option 'rebalance-cmd' for volume 'distrep2-dht' with value '5'
>[2012-09-21 06:57:08.363176] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-dht: adding option 'assert-no-child-down' for volume 'distrep2-dht' with value 'yes'
>[2012-09-21 06:57:08.363187] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-dht: adding option 'lookup-unhashed' for volume 'distrep2-dht' with value 'yes'
>[2012-09-21 06:57:08.363197] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-dht: adding option 'use-readdirp' for volume 'distrep2-dht' with value 'yes'
>[2012-09-21 06:57:08.363208] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-2: adding option 'entry-self-heal' for volume 'distrep2-replicate-2' with value 'off'
>[2012-09-21 06:57:08.363219] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-2: adding option 'metadata-self-heal' for volume 'distrep2-replicate-2' with value 'off'
>[2012-09-21 06:57:08.363230] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-2: adding option 'data-self-heal' for volume 'distrep2-replicate-2' with value 'off'
>[2012-09-21 06:57:08.363240] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-1: adding option 'entry-self-heal' for volume 'distrep2-replicate-1' with value 'off'
>[2012-09-21 06:57:08.363250] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-1: adding option 'metadata-self-heal' for volume 'distrep2-replicate-1' with value 'off'
>[2012-09-21 06:57:08.363260] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-1: adding option 'data-self-heal' for volume 'distrep2-replicate-1' with value 'off'
>[2012-09-21 06:57:08.363271] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-0: adding option 'entry-self-heal' for volume 'distrep2-replicate-0' with value 'off'
>[2012-09-21 06:57:08.363281] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-0: adding option 'metadata-self-heal' for volume 'distrep2-replicate-0' with value 'off'
>[2012-09-21 06:57:08.363300] I [graph.c:241:gf_add_cmdline_options] 0-distrep2-replicate-0: adding option 'data-self-heal' for volume 'distrep2-replicate-0' with value 'off'
>[2012-09-21 06:57:08.374499] I [client.c:2142:notify] 0-distrep2-client-0: parent translators are ready, attempting connect on transport
>[2012-09-21 06:57:08.692509] I [client.c:2142:notify] 0-distrep2-client-1: parent translators are ready, attempting connect on transport
>[2012-09-21 06:57:08.696265] I [client.c:2142:notify] 0-distrep2-client-2: parent translators are ready, attempting connect on transport
>[2012-09-21 06:57:09.013611] I [client.c:2142:notify] 0-distrep2-client-3: parent translators are ready, attempting connect on transport
>[2012-09-21 06:57:09.481053] I [client.c:2142:notify] 0-distrep2-client-4: parent translators are ready, attempting connect on transport
>[2012-09-21 06:57:09.484967] I [client.c:2142:notify] 0-distrep2-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume distrep2-client-0
>  2:     type protocol/client
>  3:     option remote-host rhs-gp-srv4.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /disk2
>  5:     option transport-type tcp
>  6:     option username c1e5cd8f-f576-4b4d-8997-11783705a5fe
>  7:     option password 38424ed9-a58c-44be-a36d-ae9fef1900f4
>  8: end-volume
>  9: 
> 10: volume distrep2-client-1
> 11:     type protocol/client
> 12:     option remote-host rhs-gp-srv11.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /disk2
> 14:     option transport-type tcp
> 15:     option username c1e5cd8f-f576-4b4d-8997-11783705a5fe
> 16:     option password 38424ed9-a58c-44be-a36d-ae9fef1900f4
> 17: end-volume
> 18: 
> 19: volume distrep2-client-2
> 20:     type protocol/client
> 21:     option remote-host rhs-gp-srv12.lab.eng.blr.redhat.com
> 22:     option remote-subvolume /disk2
> 23:     option transport-type tcp
> 24:     option username c1e5cd8f-f576-4b4d-8997-11783705a5fe
> 25:     option password 38424ed9-a58c-44be-a36d-ae9fef1900f4
> 26: end-volume
> 27: 
> 28: volume distrep2-client-3
> 29:     type protocol/client
> 30:     option remote-host rhs-gp-srv15.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /disk2
> 32:     option transport-type tcp
> 33:     option username c1e5cd8f-f576-4b4d-8997-11783705a5fe
> 34:     option password 38424ed9-a58c-44be-a36d-ae9fef1900f4
> 35: end-volume
> 36: 
> 37: volume distrep2-client-4
> 38:     type protocol/client
> 39:     option remote-host rhs-gp-srv1.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /disk2/brick1
> 41:     option transport-type tcp
> 42:     option username c1e5cd8f-f576-4b4d-8997-11783705a5fe
> 43:     option password 38424ed9-a58c-44be-a36d-ae9fef1900f4
> 44: end-volume
> 45: 
> 46: volume distrep2-client-5
> 47:     type protocol/client
> 48:     option remote-host rhs-gp-srv2.lab.eng.blr.redhat.com
> 49:     option remote-subvolume /disk2/brick1
> 50:     option transport-type tcp
> 51:     option username c1e5cd8f-f576-4b4d-8997-11783705a5fe
> 52:     option password 38424ed9-a58c-44be-a36d-ae9fef1900f4
> 53: end-volume
> 54: 
> 55: volume distrep2-replicate-0
> 56:     type cluster/replicate
> 57:     option eager-lock enable
> 58:     subvolumes distrep2-client-0 distrep2-client-1
> 59: end-volume
> 60: 
> 61: volume distrep2-replicate-1
> 62:     type cluster/replicate
> 63:     option eager-lock enable
> 64:     subvolumes distrep2-client-2 distrep2-client-3
> 65: end-volume
> 66: 
> 67: volume distrep2-replicate-2
> 68:     type cluster/replicate
> 69:     option eager-lock enable
> 70:     subvolumes distrep2-client-4 distrep2-client-5
> 71: end-volume
> 72: 
> 73: volume distrep2-dht
> 74:     type cluster/distribute
> 75:     subvolumes distrep2-replicate-0 distrep2-replicate-1 distrep2-replicate-2
> 76: end-volume
> 77: 
> 78: volume distrep2-write-behind
> 79:     type performance/write-behind
> 80:     subvolumes distrep2-dht
> 81: end-volume
> 82: 
> 83: volume distrep2
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes distrep2-write-behind
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-09-21 06:57:09.489593] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-distrep2-client-0: changing port to 24009 (from 0)
>[2012-09-21 06:57:13.368253] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-distrep2-client-2: changing port to 24009 (from 0)
>[2012-09-21 06:57:13.375465] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-distrep2-client-4: changing port to 24011 (from 0)
>[2012-09-21 06:57:13.375743] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-0: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:57:13.375836] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-distrep2-client-3: changing port to 24009 (from 0)
>[2012-09-21 06:57:13.375871] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-distrep2-client-5: changing port to 24011 (from 0)
>[2012-09-21 06:57:13.375979] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-0: Connected to 10.70.36.8:24009, attached to remote volume '/disk2'.
>[2012-09-21 06:57:13.375997] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:57:13.376064] I [afr-common.c:3631:afr_notify] 0-distrep2-replicate-0: Subvolume 'distrep2-client-0' came back up; going online.
>[2012-09-21 06:57:13.376097] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-0: Server lk version = 1
>[2012-09-21 06:57:13.379999] I [rpc-clnt.c:1659:rpc_clnt_reconfig] 0-distrep2-client-1: changing port to 24009 (from 0)
>[2012-09-21 06:57:17.377828] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-2: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:57:17.378260] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-2: Connected to 10.70.36.16:24009, attached to remote volume '/disk2'.
>[2012-09-21 06:57:17.378285] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:57:17.378344] I [afr-common.c:3631:afr_notify] 0-distrep2-replicate-1: Subvolume 'distrep2-client-2' came back up; going online.
>[2012-09-21 06:57:17.378515] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-2: Server lk version = 1
>[2012-09-21 06:57:17.381474] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-4: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:57:17.381822] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-4: Connected to 10.70.36.5:24011, attached to remote volume '/disk2/brick1'.
>[2012-09-21 06:57:17.381841] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:57:17.381889] I [afr-common.c:3631:afr_notify] 0-distrep2-replicate-2: Subvolume 'distrep2-client-4' came back up; going online.
>[2012-09-21 06:57:17.382069] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-4: Server lk version = 1
>[2012-09-21 06:57:17.385472] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-3: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:57:17.385821] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-3: Connected to 10.70.36.19:24009, attached to remote volume '/disk2'.
>[2012-09-21 06:57:17.385847] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:57:17.386085] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-3: Server lk version = 1
>[2012-09-21 06:57:17.389425] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-5: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:57:17.389726] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-5: Connected to 10.70.36.6:24011, attached to remote volume '/disk2/brick1'.
>[2012-09-21 06:57:17.389766] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:57:17.390004] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-5: Server lk version = 1
>[2012-09-21 06:57:17.393198] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-1: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:57:17.393462] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-1: Connected to 10.70.36.15:24009, attached to remote volume '/disk2'.
>[2012-09-21 06:57:17.393480] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:57:17.393679] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-1: Server lk version = 1
>[2012-09-21 06:57:17.399698] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-distrep2-replicate-0: added root inode
>[2012-09-21 06:57:17.400229] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-distrep2-replicate-1: added root inode
>[2012-09-21 06:57:17.400332] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-distrep2-replicate-2: added root inode
>[2012-09-21 06:57:17.402007] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 06:57:17.413631] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 06:57:17.422822] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:57:17.425525] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:57:17.432664] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:57:17.435182] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:57:17.440042] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:57:20.205992] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:57:20.206021] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:57:32.421302] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:57:32.421355] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:57:33.902567] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:57:33.902602] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:57:35.336217] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:57:35.336250] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:57:37.474678] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:57:37.474731] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:58:21.670225] W [socket.c:1512:__socket_proto_state_machine] 0-distrep2-client-5: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.36.6:24011)
>[2012-09-21 06:58:21.670341] I [client.c:2090:client_rpc_notify] 0-distrep2-client-5: disconnected
>[2012-09-21 06:58:27.903137] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:58:27.903200] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:58:29.418836] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:58:29.418877] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:58:30.823745] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:58:30.823777] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:58:32.404652] E [socket.c:1715:socket_connect_finish] 0-distrep2-client-5: connection to 10.70.36.6:24011 failed (Connection refused)
>[2012-09-21 06:58:41.413347] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-distrep2-client-1: server 10.70.36.15:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-21 06:58:41.414198] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-09-21 06:57:17.447626 (xid=0x57x)
>[2012-09-21 06:58:41.414240] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 06:58:41.414336] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-09-21 06:57:59.397211 (xid=0x58x)
>[2012-09-21 06:58:41.414355] W [client-handshake.c:275:client_ping_cbk] 0-distrep2-client-1: timer must have expired
>[2012-09-21 06:58:41.414381] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 06:58:21.675853 (xid=0x59x)
>[2012-09-21 06:58:41.414402] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 06:58:41.420383] I [socket.c:2315:socket_submit_request] 0-distrep2-client-1: not connected (priv->connected = 0)
>[2012-09-21 06:58:41.420414] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x60x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (distrep2-client-1)
>[2012-09-21 06:58:41.420431] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 06:58:41.420700] I [client.c:2090:client_rpc_notify] 0-distrep2-client-1: disconnected
>[2012-09-21 06:58:41.421062] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-1: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:58:41.421330] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-1: Connected to 10.70.36.15:24009, attached to remote volume '/disk2'.
>[2012-09-21 06:58:41.421347] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:58:41.421357] I [client-handshake.c:1282:client_post_handshake] 0-distrep2-client-1: 4 fds open - Delaying child_up until they are re-opened
>[2012-09-21 06:58:41.421450] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:58:41.421549] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 06:58:41.421582] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:58:41.421651] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:58:41.421698] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 06:58:41.421713] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:58:41.421752] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:58:41.421788] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 06:58:41.421802] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:58:41.421845] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 0)
>[2012-09-21 06:58:41.421866] I [client-lk.c:601:decrement_reopen_fd_count] 0-distrep2-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-09-21 06:58:41.423155] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-1: Server lk version = 1
>[2012-09-21 06:59:17.906067] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:17.906098] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:19.378847] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:19.378890] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:21.588628] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:21.588657] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:22.892458] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:22.892483] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:24.169978] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:24.170004] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:25.538289] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:25.538318] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:26.703407] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:26.703436] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:27.825192] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:27.825219] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:28.848561] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:28.848587] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:30.100279] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:30.100306] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:31.695867] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:31.695901] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:33.560028] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:33.560058] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:35.244212] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:35.244239] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:36.700955] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 06:59:36.700981] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0
>[2012-09-21 06:59:46.503277] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-distrep2-client-0: server 10.70.36.8:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-21 06:59:46.503509] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-0: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-09-21 06:58:41.414299 (xid=0x58x)
>[2012-09-21 06:59:46.503543] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-0: remote operation failed: Transport endpoint is not connected
>[2012-09-21 06:59:46.503600] E [dht-rebalance.c:721:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: failed to migrate data
>[2012-09-21 06:59:46.503601] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 06:58:41.414517 (xid=0x59x)
>[2012-09-21 06:59:46.503728] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 06:59:46.503790] I [dht-layout.c:593:dht_layout_normalize] 0-distrep2-dht: found anomalies in /. holes=1 overlaps=0
>[2012-09-21 06:59:46.503804] W [dht-selfheal.c:875:dht_selfheal_directory] 0-distrep2-dht: 1 subvolumes down -- not fixing
>[2012-09-21 06:59:46.503877] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 06:58:41.426965 (xid=0x60x)
>[2012-09-21 06:59:46.503890] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 06:59:46.503896] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 06:59:46.508085] I [socket.c:2315:socket_submit_request] 0-distrep2-client-0: not connected (priv->connected = 0)
>[2012-09-21 06:59:46.508112] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-0: failed to submit rpc-request (XID: 0x63x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (distrep2-client-0)
>[2012-09-21 06:59:46.508132] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 06:59:46.508155] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-0: failed to submit rpc-request (XID: 0x64x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (distrep2-client-0)
>[2012-09-21 06:59:46.508194] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-0: remote operation failed: Transport endpoint is not connected
>[2012-09-21 06:59:46.508353] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 06:58:41.429899 (xid=0x61x)
>[2012-09-21 06:59:46.508375] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 06:59:46.508426] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-0: failed to submit rpc-request (XID: 0x65x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (distrep2-client-0)
>[2012-09-21 06:59:46.508444] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 06:59:46.508672] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-0: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-09-21 06:59:04.445781 (xid=0x62x)
>[2012-09-21 06:59:46.508698] W [client-handshake.c:275:client_ping_cbk] 0-distrep2-client-0: timer must have expired
>[2012-09-21 06:59:46.508712] I [client.c:2090:client_rpc_notify] 0-distrep2-client-0: disconnected
>[2012-09-21 06:59:46.509104] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-0: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:59:46.509221] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 06:59:46.509371] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 06:59:46.509429] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-0: Connected to 10.70.36.8:24009, attached to remote volume '/disk2'.
>[2012-09-21 06:59:46.509468] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:59:46.509484] I [client-handshake.c:1282:client_post_handshake] 0-distrep2-client-0: 4 fds open - Delaying child_up until they are re-opened
>[2012-09-21 06:59:46.509570] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:59:46.509636] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-0: invalid inode
>[2012-09-21 06:59:46.509655] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-0: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:59:46.509694] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:59:46.509732] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-0: invalid inode
>[2012-09-21 06:59:46.509746] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-0: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:59:46.509782] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:59:46.509819] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-0: invalid inode
>[2012-09-21 06:59:46.509832] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-0: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:59:46.509891] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-0: reopendir on / succeeded (fd = 0)
>[2012-09-21 06:59:46.509907] I [client-lk.c:601:decrement_reopen_fd_count] 0-distrep2-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-09-21 06:59:46.510810] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-0: Server lk version = 1
>[2012-09-21 06:59:46.514773] E [dht-rebalance.c:1202:gf_defrag_migrate_data] 0-distrep2-dht: migrate-data failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata
>[2012-09-21 06:59:46.516528] I [dht-layout.c:593:dht_layout_normalize] 0-distrep2-dht: found anomalies in /. holes=2 overlaps=1
>[2012-09-21 06:59:46.516716] I [dht-layout.c:593:dht_layout_normalize] 0-distrep2-dht: found anomalies in /. holes=2 overlaps=1
>[2012-09-21 06:59:46.517009] I [dht-layout.c:593:dht_layout_normalize] 0-distrep2-dht: found anomalies in /. holes=2 overlaps=1
>[2012-09-21 06:59:46.517575] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-distrep2-client-5: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/outbox (00000000-0000-0000-0000-000000000000). Key: trusted.glusterfs.pathinfo
>[2012-09-21 06:59:46.517638] I [dht-layout.c:593:dht_layout_normalize] 0-distrep2-dht: found anomalies in /. holes=2 overlaps=1
>[2012-09-21 06:59:46.518331] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/outbox: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:59:46.518346] I [dht-layout.c:593:dht_layout_normalize] 0-distrep2-dht: found anomalies in /. holes=2 overlaps=1
>[2012-09-21 06:59:46.527294] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 06:59:46.529703] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 06:59:46.531398] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 06:59:46.532779] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 06:59:46.535274] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 06:59:46.537301] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.540032] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 06:59:46.541268] I [dht-rebalance.c:856:dht_migrate_file] 0-distrep2-dht: completed migration of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/outbox from subvolume distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:59:46.541332] W [client3_1-fops.c:5628:client3_1_readdirp] 0-distrep2-client-1:  (00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD
>[2012-09-21 06:59:46.541731] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.541779] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.541985] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 06:59:46.543413] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.544586] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.546894] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.550646] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.551788] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.552423] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.553201] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 06:59:46.553729] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.555058] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.556733] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.556892] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 06:59:46.557150] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.557461] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 06:59:46.559278] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.560711] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.563609] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.564138] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-distrep2-client-5: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata (00000000-0000-0000-0000-000000000000). Key: trusted.glusterfs.pathinfo
>[2012-09-21 06:59:46.564382] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:59:46.566344] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.566461] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-distrep2-client-5: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata (00000000-0000-0000-0000-000000000000). Key: trusted.glusterfs.pathinfo
>[2012-09-21 06:59:46.566670] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.568811] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:59:46.569275] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-distrep2-client-5: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata (00000000-0000-0000-0000-000000000000). Key: trusted.glusterfs.pathinfo
>[2012-09-21 06:59:46.569362] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 06:59:46.569654] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:59:46.570809] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8
>[2012-09-21 06:59:46.573237] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8
>[2012-09-21 06:59:46.573319] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-distrep2-client-5: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata (00000000-0000-0000-0000-000000000000). Key: trusted.glusterfs.pathinfo
>[2012-09-21 06:59:46.573582] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:59:46.573990] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-0: remote operation failed: Invalid argument
>[2012-09-21 06:59:46.575095] W [client3_1-fops.c:1059:client3_1_getxattr_cbk] 0-distrep2-client-5: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata (00000000-0000-0000-0000-000000000000). Key: trusted.glusterfs.pathinfo
>[2012-09-21 06:59:46.575262] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 06:59:53.516667] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-5: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 06:59:53.517083] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-5: Connected to 10.70.36.6:24011, attached to remote volume '/disk2/brick1'.
>[2012-09-21 06:59:53.517110] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 06:59:53.517121] I [client-handshake.c:1282:client_post_handshake] 0-distrep2-client-5: 4 fds open - Delaying child_up until they are re-opened
>[2012-09-21 06:59:53.517224] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:59:53.517269] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-5: invalid inode
>[2012-09-21 06:59:53.517285] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-5: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:59:53.517319] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:59:53.517353] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-5: invalid inode
>[2012-09-21 06:59:53.517366] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-5: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:59:53.517397] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 06:59:53.517429] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-5: invalid inode
>[2012-09-21 06:59:53.517443] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-5: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 06:59:53.517656] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-5: reopendir on / succeeded (fd = 0)
>[2012-09-21 06:59:53.517678] I [client-lk.c:601:decrement_reopen_fd_count] 0-distrep2-client-5: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-09-21 06:59:53.517892] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-5: Server lk version = 1
>[2012-09-21 07:00:12.104666] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:12.104700] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:00:17.517616] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:17.517649] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:00:19.842885] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:19.842910] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:00:21.213458] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:21.213485] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:00:22.705214] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:22.705242] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:00:25.138285] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:25.138314] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:00:26.726470] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:26.726497] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:00:39.019382] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:00:39.019411] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:01:03.525477] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:01:03.525509] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:01:05.764993] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:01:05.765020] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:01:07.162680] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:01:07.162709] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:01:08.512967] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:01:08.512994] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 17, failures: 1
>[2012-09-21 07:01:10.526936] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-distrep2-client-1: server 10.70.36.15:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-21 07:01:10.527129] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-09-21 06:59:46.574034 (xid=0x363x)
>[2012-09-21 07:01:10.527168] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:01:10.527218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(SETXATTR(17)) called at 2012-09-21 06:59:46.574489 (xid=0x366x)
>[2012-09-21 07:01:10.527252] W [client3_1-fops.c:992:client3_1_setxattr_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:01:10.527275] E [dht-rebalance.c:721:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: failed to migrate data
>[2012-09-21 07:01:10.531314] I [socket.c:2315:socket_submit_request] 0-distrep2-client-1: not connected (priv->connected = 0)
>[2012-09-21 07:01:10.531345] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x373x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:01:10.531378] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:01:10.531426] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(READDIR(28)) called at 2012-09-21 06:59:46.574516 (xid=0x367x)
>[2012-09-21 07:01:10.531443] W [client3_1-fops.c:2250:client3_1_readdir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected remote_fd = 20
>[2012-09-21 07:01:10.531453] I [afr-dir-read.c:117:afr_examine_dir_readdir_cbk] 0-distrep2-replicate-0: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8: failed to do opendir on distrep2-client-1
>[2012-09-21 07:01:10.531501] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2012-09-21 06:59:46.574574 (xid=0x368x)
>[2012-09-21 07:01:10.531519] W [client3_1-fops.c:418:client3_1_open_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:01:10.531580] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 06:59:46.575343 (xid=0x369x)
>[2012-09-21 07:01:10.531598] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:01:10.531625] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x374x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:01:10.531663] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:01:10.531672] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2012-09-21 06:59:46.576251 (xid=0x370x)
>[2012-09-21 07:01:10.531704] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:01:10.531745] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 06:59:53.524024 (xid=0x371x)
>[2012-09-21 07:01:10.531761] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 07:01:10.531844] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x375x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:01:10.531864] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 07:01:10.532065] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-09-21 07:00:28.521070 (xid=0x372x)
>[2012-09-21 07:01:10.532099] W [client-handshake.c:275:client_ping_cbk] 0-distrep2-client-1: timer must have expired
>[2012-09-21 07:01:10.532119] I [client.c:2090:client_rpc_notify] 0-distrep2-client-1: disconnected
>[2012-09-21 07:01:10.532529] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-1: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 07:01:10.532806] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 07:01:10.532848] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-1: Connected to 10.70.36.15:24009, attached to remote volume '/disk2'.
>[2012-09-21 07:01:10.532869] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 07:01:10.532879] I [client-handshake.c:1282:client_post_handshake] 0-distrep2-client-1: 21 fds open - Delaying child_up until they are re-opened
>[2012-09-21 07:01:10.532959] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533005] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533019] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533172] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533227] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533264] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533322] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533380] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533397] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533430] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533476] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533491] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533522] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533555] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533570] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533601] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533640] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533655] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533686] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533726] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533739] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533769] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533802] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533816] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533845] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533876] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533890] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533918] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.533949] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.533962] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.533991] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.534022] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.534040] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.534069] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.534100] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.534113] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.534141] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.534173] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.534186] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.534215] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.534245] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.534259] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.534287] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:01:10.534318] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:01:10.534336] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:01:10.534574] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 0)
>[2012-09-21 07:01:10.534665] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 1)
>[2012-09-21 07:01:10.534737] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 2)
>[2012-09-21 07:01:10.534766] E [dht-rebalance.c:1202:gf_defrag_migrate_data] 0-distrep2-dht: migrate-data failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata
>[2012-09-21 07:01:10.534820] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 3)
>[2012-09-21 07:01:10.534897] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 4)
>[2012-09-21 07:01:10.534950] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 5)
>[2012-09-21 07:01:10.534969] I [client-lk.c:601:decrement_reopen_fd_count] 0-distrep2-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-09-21 07:01:10.535184] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-1: Server lk version = 1
>[2012-09-21 07:01:10.538512] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8/86c8e2c0-3a32-4f00-ad21-1c02afc304db: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 07:01:10.538777] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 07:01:10.538958] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 07:01:10.543470] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 07:01:10.544595] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 07:01:10.546165] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 07:01:10.548911] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:01:10.550319] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:01:10.614696] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:01:10.615859] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:01:10.616022] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:01:10.617653] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:01:10.618789] W [client3_1-fops.c:4168:client3_1_fstat] 0-distrep2-client-1:  (00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD
>[2012-09-21 07:01:10.619115] W [client3_1-fops.c:3903:client3_1_readv] 0-distrep2-client-1:  (00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD
>[2012-09-21 07:01:10.619217] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:01:10.619620] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-0: remote operation failed: Invalid argument
>[2012-09-21 07:01:10.619687] E [dht-rebalance.c:721:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: failed to migrate data
>[2012-09-21 07:01:10.620866] E [dht-rebalance.c:1202:gf_defrag_migrate_data] 0-distrep2-dht: migrate-data failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata
>[2012-09-21 07:01:10.620924] W [client3_1-fops.c:5628:client3_1_readdirp] 0-distrep2-client-1:  (00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD
>[2012-09-21 07:01:10.621368] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:01:10.625279] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 07:01:10.626282] W [client3_1-fops.c:1444:client3_1_fstat_cbk] 0-distrep2-client-1: remote operation failed: No such file or directory
>[2012-09-21 07:01:10.627495] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-1: remote operation failed: No such file or directory
>[2012-09-21 07:01:10.628457] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-0: remote operation failed: Invalid argument
>[2012-09-21 07:01:10.628494] E [dht-rebalance.c:721:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: failed to migrate data
>[2012-09-21 07:01:10.628656] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-0: remote operation failed: Invalid argument
>[2012-09-21 07:01:10.629321] E [dht-rebalance.c:1202:gf_defrag_migrate_data] 0-distrep2-dht: migrate-data failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata
>[2012-09-21 07:01:10.630642] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 07:01:10.630929] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-1: remote operation failed: No such file or directory
>[2012-09-21 07:01:10.630967] E [dht-rebalance.c:721:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: failed to migrate data
>[2012-09-21 07:01:10.631365] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8
>[2012-09-21 07:01:10.631597] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 07:01:10.633001] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 07:01:10.633702] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-0: remote operation failed: Invalid argument
>[2012-09-21 07:01:10.633814] E [dht-rebalance.c:1202:gf_defrag_migrate_data] 0-distrep2-dht: migrate-data failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata
>[2012-09-21 07:01:10.633868] W [client3_1-fops.c:5628:client3_1_readdirp] 0-distrep2-client-1:  (00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD
>[2012-09-21 07:01:10.634648] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8
>[2012-09-21 07:02:35.534964] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-distrep2-client-1: server 10.70.36.15:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-21 07:02:35.535172] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(INODELK(29)) called at 2012-09-21 07:01:10.634020 (xid=0x528x)
>[2012-09-21 07:02:35.535240] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:02:35.535282] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(INODELK(29)) called at 2012-09-21 07:01:10.635022 (xid=0x531x)
>[2012-09-21 07:02:35.535316] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:02:35.535413] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-09-21 07:01:10.633763 (xid=0x527x)
>[2012-09-21 07:02:35.535431] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:02:35.535458] E [dht-rebalance.c:721:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: failed to migrate data
>[2012-09-21 07:02:35.535489] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 07:01:10.634124 (xid=0x529x)
>[2012-09-21 07:02:35.535520] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:02:35.535612] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 07:02:35.535660] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(OPENDIR(20)) called at 2012-09-21 07:01:10.634719 (xid=0x530x)
>[2012-09-21 07:02:35.535684] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8 (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:02:35.541154] I [socket.c:2315:socket_submit_request] 0-distrep2-client-1: not connected (priv->connected = 0)
>[2012-09-21 07:02:35.541185] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x538x Program: GlusterFS 3.1, ProgVers: 330, Proc: 20) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:02:35.541212] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:02:35.541213] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x539x Program: GlusterFS 3.1, ProgVers: 330, Proc: 28) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:02:35.541261] W [client3_1-fops.c:2250:client3_1_readdir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected remote_fd = -2
>[2012-09-21 07:02:35.541272] I [afr-dir-read.c:117:afr_examine_dir_readdir_cbk] 0-distrep2-replicate-0: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8: failed to do opendir on distrep2-client-1
>[2012-09-21 07:02:35.541303] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 07:01:10.635216 (xid=0x532x)
>[2012-09-21 07:02:35.541328] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/leases (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:02:35.541391] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(RELEASEDIR(42)) called at 2012-09-21 07:01:10.635549 (xid=0x533x)
>[2012-09-21 07:02:35.541430] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 07:01:10.636150 (xid=0x534x)
>[2012-09-21 07:02:35.541447] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:02:35.541554] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(OPENDIR(20)) called at 2012-09-21 07:01:10.636847 (xid=0x535x)
>[2012-09-21 07:02:35.541573] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:02:35.541581] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:02:35.541675] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(OPENDIR(20)) called at 2012-09-21 07:01:10.638416 (xid=0x536x)
>[2012-09-21 07:02:35.541702] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:02:35.541754] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x540x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:02:35.541754] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-09-21 07:01:53.530926 (xid=0x537x)
>[2012-09-21 07:02:35.541797] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x541x Program: GlusterFS 3.1, ProgVers: 330, Proc: 40) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:02:35.541784] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:02:35.541833] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:02:35.541842] W [client-handshake.c:275:client_ping_cbk] 0-distrep2-client-1: timer must have expired
>[2012-09-21 07:02:35.541891] I [client.c:2090:client_rpc_notify] 0-distrep2-client-1: disconnected
>[2012-09-21 07:02:35.542977] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-1: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 07:02:35.544143] E [client3_1-fops.c:2229:client3_1_readdir_cbk] 0-distrep2-client-1: XDR decoding failed
>[2012-09-21 07:02:35.544166] W [client3_1-fops.c:2250:client3_1_readdir_cbk] 0-distrep2-client-1: remote operation failed: Invalid argument remote_fd = -2
>[2012-09-21 07:02:35.544177] I [afr-dir-read.c:117:afr_examine_dir_readdir_cbk] 0-distrep2-replicate-0: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md: failed to do opendir on distrep2-client-1
>[2012-09-21 07:02:35.544270] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-1: Connected to 10.70.36.15:24009, attached to remote volume '/disk2'.
>[2012-09-21 07:02:35.544288] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 07:02:35.544298] I [client-handshake.c:1282:client_post_handshake] 0-distrep2-client-1: 16 fds open - Delaying child_up until they are re-opened
>[2012-09-21 07:02:35.544430] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:02:35.544488] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:02:35.544511] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:02:35.544555] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:02:35.544597] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:02:35.544618] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:02:35.544729] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:02:35.544775] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:02:35.544809] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:02:35.544859] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:02:35.544927] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:02:35.544944] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:02:35.544982] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:02:35.545021] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:02:35.545036] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:02:35.545067] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:02:35.545100] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:02:35.545113] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:02:35.545143] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:02:35.545175] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:02:35.545188] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:02:35.545344] E [dht-rebalance.c:1202:gf_defrag_migrate_data] 0-distrep2-dht: migrate-data failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata
>[2012-09-21 07:02:35.545485] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 0)
>[2012-09-21 07:02:35.545562] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 1)
>[2012-09-21 07:02:35.545705] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 2)
>[2012-09-21 07:02:35.545853] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 3)
>[2012-09-21 07:02:35.545960] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 4)
>[2012-09-21 07:02:35.546297] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 5)
>[2012-09-21 07:02:35.546448] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 6)
>[2012-09-21 07:02:35.546713] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 8)
>[2012-09-21 07:02:35.547088] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 7)
>[2012-09-21 07:02:35.547116] I [client-lk.c:601:decrement_reopen_fd_count] 0-distrep2-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-09-21 07:02:35.547825] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-1: Server lk version = 1
>[2012-09-21 07:02:35.547923] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md
>[2012-09-21 07:02:35.549049] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:02:35.549406] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 07:02:35.550228] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 07:02:35.550672] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:02:35.552001] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8/86c8e2c0-3a32-4f00-ad21-1c02afc304db: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 07:02:35.553138] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:02:35.567120] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 07:02:35.567132] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 07:02:35.567520] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 07:02:35.568220] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:02:35.570192] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:02:35.570462] I [dht-rebalance.c:647:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: attempting to move from distrep2-replicate-0 to distrep2-replicate-2
>[2012-09-21 07:02:35.571982] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-distrep2-dht: migrate data called on /
>[2012-09-21 07:02:35.577409] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-0: remote operation failed: Invalid argument
>[2012-09-21 07:02:38.093664] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:02:38.093719] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 44, failures: 6
>[2012-09-21 07:02:40.404967] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:02:40.405008] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 44, failures: 6
>[2012-09-21 07:02:42.028243] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:02:42.028274] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 44, failures: 6
>[2012-09-21 07:02:43.518802] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:02:43.518831] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 44, failures: 6
>[2012-09-21 07:03:43.636789] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:03:43.636852] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 44, failures: 6
>[2012-09-21 07:03:45.269981] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:03:45.270012] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 44, failures: 6
>[2012-09-21 07:03:46.545301] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress
>[2012-09-21 07:03:46.545333] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 44, failures: 6
>[2012-09-21 07:04:00.544543] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-distrep2-client-1: server 10.70.36.15:24009 has not responded in the last 42 seconds, disconnecting.
>[2012-09-21 07:04:00.544761] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2012-09-21 07:02:35.577465 (xid=0x630x)
>[2012-09-21 07:04:00.544817] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.544853] E [dht-rebalance.c:721:dht_migrate_file] 0-distrep2-dht: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata: failed to migrate data
>[2012-09-21 07:04:00.544868] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 07:02:35.577669 (xid=0x632x)
>[2012-09-21 07:04:00.544919] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679 (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:04:00.549043] I [socket.c:2315:socket_submit_request] 0-distrep2-client-1: not connected (priv->connected = 0)
>[2012-09-21 07:04:00.549073] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x645x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:04:00.549096] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679 (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:04:00.549347] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(SETXATTR(17)) called at 2012-09-21 07:02:35.577856 (xid=0x633x)
>[2012-09-21 07:04:00.549378] W [client3_1-fops.c:992:client3_1_setxattr_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.549452] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x646x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:04:00.549470] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.549507] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2012-09-21 07:02:35.578093 (xid=0x634x)
>[2012-09-21 07:04:00.549523] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.549558] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 07:02:35.578535 (xid=0x635x)
>[2012-09-21 07:04:00.549574] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:04:00.549644] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:04:00.549664] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2012-09-21 07:02:35.579693 (xid=0x636x)
>[2012-09-21 07:04:00.549710] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8 (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:04:00.549778] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8
>[2012-09-21 07:04:00.549787] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(RELEASEDIR(42)) called at 2012-09-21 07:02:35.579974 (xid=0x637x)
>[2012-09-21 07:04:00.549808] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x647x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:04:00.549845] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(OPENDIR(20)) called at 2012-09-21 07:02:35.580056 (xid=0x638x)
>[2012-09-21 07:04:00.549861] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.549874] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2012-09-21 07:04:00.549896] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x648x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:04:00.549937] W [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.549941] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(RELEASEDIR(42)) called at 2012-09-21 07:02:35.582672 (xid=0x639x)
>[2012-09-21 07:04:00.549975] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x649x Program: GlusterFS 3.1, ProgVers: 330, Proc: 40) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:04:00.550000] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(RELEASEDIR(42)) called at 2012-09-21 07:02:35.582699 (xid=0x640x)
>[2012-09-21 07:02:35.550015] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.550047] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(OPENDIR(20)) called at 2012-09-21 07:02:35.582840 (xid=0x641x)
>[2012-09-21 07:04:00.550076] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:04:00.550112] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(OPENDIR(20)) called at 2012-09-21 07:02:35.582909 (xid=0x642x)
>[2012-09-21 07:04:00.550128] W [client3_1-fops.c:2546:client3_1_opendir_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected. Path: /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images (00000000-0000-0000-0000-000000000000)
>[2012-09-21 07:04:00.550161] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2012-09-21 07:02:43.537394 (xid=0x643x)
>[2012-09-21 07:04:00.550175] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.550176] W [rpc-clnt.c:1498:rpc_clnt_submit] 0-distrep2-client-1: failed to submit rpc-request (XID: 0x650x Program: GlusterFS 3.1, ProgVers: 330, Proc: 40) to rpc-transport (distrep2-client-1)
>[2012-09-21 07:04:00.550208] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-distrep2-client-1: remote operation failed: Transport endpoint is not connected
>[2012-09-21 07:04:00.550209] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f411e5fc818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f411e5fc4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f411e5fbf3e]))) 0-distrep2-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-09-21 07:03:18.539676 (xid=0x644x)
>[2012-09-21 07:04:00.550254] W [client-handshake.c:275:client_ping_cbk] 0-distrep2-client-1: timer must have expired
>[2012-09-21 07:04:00.550268] I [client.c:2090:client_rpc_notify] 0-distrep2-client-1: disconnected
>[2012-09-21 07:04:00.551184] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:04:00.551268] I [client-handshake.c:1636:select_server_supported_programs] 0-distrep2-client-1: Using Program GlusterFS 3.3.0rhsvirt1, Num (1298437), Version (330)
>[2012-09-21 07:04:00.551928] I [client-handshake.c:1433:client_setvolume_cbk] 0-distrep2-client-1: Connected to 10.70.36.15:24009, attached to remote volume '/disk2'.
>[2012-09-21 07:04:00.551949] I [client-handshake.c:1445:client_setvolume_cbk] 0-distrep2-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-09-21 07:04:00.551959] I [client-handshake.c:1282:client_post_handshake] 0-distrep2-client-1: 14 fds open - Delaying child_up until they are re-opened
>[2012-09-21 07:04:00.552174] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:04:00.552238] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:04:00.552262] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:04:00.552309] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:04:00.552362] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:04:00.552390] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:04:00.552452] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:04:00.552500] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x145) [0x7f411a2b3fa5] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopen+0xd7) [0x7f411a2b3657] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:04:00.552515] W [client-handshake.c:1187:protocol_client_reopen] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:04:00.552556] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-: Assertion failed: 0
>[2012-09-21 07:04:00.552592] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(client_post_handshake+0x110) [0x7f411a2b3f70] (-->/usr/lib64/glusterfs/3.3.0rhsvirt1/xlator/protocol/client.so(protocol_client_reopendir+0x107) [0x7f411a2b3be7] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f411e830b0a]))) 0-distrep2-client-1: invalid inode
>[2012-09-21 07:04:00.552605] W [client-handshake.c:1108:protocol_client_reopendir] 0-distrep2-client-1: couldn't build path from inode 00000000-0000-0000-0000-000000000000
>[2012-09-21 07:04:00.552869] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 0)
>[2012-09-21 07:04:00.553041] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 1)
>[2012-09-21 07:04:00.553151] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 3)
>[2012-09-21 07:04:00.553478] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 2)
>[2012-09-21 07:04:00.553584] E [dht-rebalance.c:1202:gf_defrag_migrate_data] 0-distrep2-dht: migrate-data failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679/dom_md/metadata
>[2012-09-21 07:04:00.553665] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 5)
>[2012-09-21 07:04:00.553816] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 6)
>[2012-09-21 07:04:00.553977] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 4)
>[2012-09-21 07:04:00.554086] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 7)
>[2012-09-21 07:04:00.554142] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 8)
>[2012-09-21 07:04:00.554308] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-distrep2-client-1: reopendir on / succeeded (fd = 9)
>[2012-09-21 07:04:00.554324] I [client-lk.c:601:decrement_reopen_fd_count] 0-distrep2-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-09-21 07:04:00.554556] I [client-handshake.c:453:client_set_lk_version_cbk] 0-distrep2-client-1: Server lk version = 1
>[2012-09-21 07:04:00.556153] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images/8179fad6-6fe8-4f19-9fdc-76592eef53d8
>[2012-09-21 07:04:00.557128] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is completed
>[2012-09-21 07:04:00.557156] I [dht-common.c:2337:dht_setxattr] 0-distrep2-dht: fixing the layout of /
>[2012-09-21 07:04:00.557159] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 0, lookups: 46, failures: 7
>[2012-09-21 07:04:00.557463] W [glusterfsd.c:906:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3910ae5ccd] (-->/lib64/libpthread.so.0() [0x39112077f1] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d2d]))) 0-: received signum (15), shutting down
>[2012-09-21 07:04:00.557624] E [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-ÄØ9: Fix layout failed for /9c17fd91-2e28-463d-b9b3-93fcd9a77679
>[2012-09-21 07:04:00.557775] I [dht-rebalance.c:1619:gf_defrag_status_get] 0-glusterfs: Rebalance is completed
>[2012-09-21 07:04:00.557778] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-ÄØ9: migrate data called on /9c17fd91-2e28-463d-b9b3-93fcd9a77679/images
>[2012-09-21 07:04:00.557810] I [dht-rebalance.c:1622:gf_defrag_status_get] 0-glusterfs: Files migrated: 26986912, size: 0, lookups: 46, failures: 7

Actions: View

Attachments on bug 859387: 615411 | 633259