Attachment 926217 Details for Bug 1129486 – mnt.log from the client

mnt.log from the client

mnt.log (text/plain), 55.65 KB, created by Scott Merrill on 2014-08-12 21:17:02 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Scott Merrill

Created: 2014-08-12 21:17:02 UTC

Size: 55.65 KB

patch

obsolete

>[2014-08-12 21:02:27.766344] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs --volfile-server=gluster1 --volfile-id=/test /mnt)
>[2014-08-12 21:02:27.769890] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-08-12 21:02:27.769912] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
>[2014-08-12 21:02:27.777300] I [dht-shared.c:311:dht_init_regex] 0-test-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-08-12 21:02:27.778506] I [socket.c:3561:socket_init] 0-test-client-1: SSL support is NOT enabled
>[2014-08-12 21:02:27.778517] I [socket.c:3576:socket_init] 0-test-client-1: using system polling thread
>[2014-08-12 21:02:27.778698] I [socket.c:3561:socket_init] 0-test-client-0: SSL support is NOT enabled
>[2014-08-12 21:02:27.778706] I [socket.c:3576:socket_init] 0-test-client-0: using system polling thread
>[2014-08-12 21:02:27.778723] I [client.c:2294:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2014-08-12 21:02:27.781312] I [client.c:2294:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host gluster1
>  4:     option remote-subvolume /bricks/brick1/brick
>  5:     option transport-type socket
>  6:     option send-gids true
>  7: end-volume
>  8: 
>  9: volume test-client-1
> 10:     type protocol/client
> 11:     option remote-host gluster2
> 12:     option remote-subvolume /bricks/brick1/brick
> 13:     option transport-type socket
> 14:     option send-gids true
> 15: end-volume
> 16: 
> 17: volume test-replicate-0
> 18:     type cluster/replicate
> 19:     subvolumes test-client-0 test-client-1
> 20: end-volume
> 21: 
> 22: volume test-dht
> 23:     type cluster/distribute
> 24:     subvolumes test-replicate-0
> 25: end-volume
> 26: 
> 27: volume test-write-behind
> 28:     type performance/write-behind
> 29:     subvolumes test-dht
> 30: end-volume
> 31: 
> 32: volume test-read-ahead
> 33:     type performance/read-ahead
> 34:     subvolumes test-write-behind
> 35: end-volume
> 36: 
> 37: volume test-io-cache
> 38:     type performance/io-cache
> 39:     subvolumes test-read-ahead
> 40: end-volume
> 41: 
> 42: volume test-quick-read
> 43:     type performance/quick-read
> 44:     subvolumes test-io-cache
> 45: end-volume
> 46: 
> 47: volume test-open-behind
> 48:     type performance/open-behind
> 49:     subvolumes test-quick-read
> 50: end-volume
> 51: 
> 52: volume test-md-cache
> 53:     type performance/md-cache
> 54:     subvolumes test-open-behind
> 55: end-volume
> 56: 
> 57: volume test
> 58:     type debug/io-stats
> 59:     option latency-measurement off
> 60:     option count-fop-hits off
> 61:     subvolumes test-md-cache
> 62: end-volume
> 63: 
>+------------------------------------------------------------------------------+
>[2014-08-12 21:02:27.784811] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-0: changing port to 49152 (from 0)
>[2014-08-12 21:02:27.787407] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-1: changing port to 49152 (from 0)
>[2014-08-12 21:02:27.790247] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:02:27.790338] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:02:27.790695] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-1: Connected to 192.168.30.108:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 21:02:27.790707] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:02:27.790778] I [afr-common.c:4131:afr_notify] 0-test-replicate-0: Subvolume 'test-client-1' came back up; going online.
>[2014-08-12 21:02:27.790806] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-0: Connected to 192.168.30.107:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 21:02:27.790827] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:02:27.793556] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 0
>[2014-08-12 21:02:27.793603] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2014-08-12 21:02:27.793616] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2014-08-12 21:02:27.793749] I [fuse-bridge.c:3914:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.13
>[2014-08-12 21:02:27.795069] I [dht-layout.c:640:dht_layout_normalize] 0-test-dht: found anomalies in /. holes=1 overlaps=0
>[2014-08-12 21:04:23.227185] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-08-12 21:04:23.232454] I [dht-shared.c:311:dht_init_regex] 1-test-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-08-12 21:04:23.237454] I [socket.c:3561:socket_init] 1-test-client-3: SSL support is NOT enabled
>[2014-08-12 21:04:23.237466] I [socket.c:3576:socket_init] 1-test-client-3: using system polling thread
>[2014-08-12 21:04:23.237775] I [socket.c:3561:socket_init] 1-test-client-2: SSL support is NOT enabled
>[2014-08-12 21:04:23.237784] I [socket.c:3576:socket_init] 1-test-client-2: using system polling thread
>[2014-08-12 21:04:23.238048] I [socket.c:3561:socket_init] 1-test-client-1: SSL support is NOT enabled
>[2014-08-12 21:04:23.238057] I [socket.c:3576:socket_init] 1-test-client-1: using system polling thread
>[2014-08-12 21:04:23.238256] I [socket.c:3561:socket_init] 1-test-client-0: SSL support is NOT enabled
>[2014-08-12 21:04:23.238266] I [socket.c:3576:socket_init] 1-test-client-0: using system polling thread
>[2014-08-12 21:04:23.238278] I [client.c:2294:notify] 1-test-client-0: parent translators are ready, attempting connect on transport
>[2014-08-12 21:04:23.241006] I [client.c:2294:notify] 1-test-client-1: parent translators are ready, attempting connect on transport
>[2014-08-12 21:04:23.246067] I [client.c:2294:notify] 1-test-client-2: parent translators are ready, attempting connect on transport
>[2014-08-12 21:04:23.248585] I [client.c:2294:notify] 1-test-client-3: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host gluster1
>  4:     option remote-subvolume /bricks/brick1/brick
>  5:     option transport-type socket
>  6:     option send-gids true
>  7: end-volume
>  8: 
>  9: volume test-client-1
> 10:     type protocol/client
> 11:     option remote-host gluster2
> 12:     option remote-subvolume /bricks/brick1/brick
> 13:     option transport-type socket
> 14:     option send-gids true
> 15: end-volume
> 16: 
> 17: volume test-replicate-0
> 18:     type cluster/replicate
> 19:     subvolumes test-client-0 test-client-1
> 20: end-volume
> 21: 
> 22: volume test-client-2
> 23:     type protocol/client
> 24:     option remote-host gluster1
> 25:     option remote-subvolume /bricks/brick2/brick
> 26:     option transport-type socket
> 27:     option send-gids true
> 28: end-volume
> 29: 
> 30: volume test-client-3
> 31:     type protocol/client
> 32:     option remote-host gluster2
> 33:     option remote-subvolume /bricks/brick2/brick
> 34:     option transport-type socket
> 35:     option send-gids true
> 36: end-volume
> 37: 
> 38: volume test-replicate-1
> 39:     type cluster/replicate
> 40:     subvolumes test-client-2 test-client-3
> 41: end-volume
> 42: 
> 43: volume test-dht
> 44:     type cluster/distribute
> 45:     subvolumes test-replicate-0 test-replicate-1
> 46: end-volume
> 47: 
> 48: volume test-write-behind
> 49:     type performance/write-behind
> 50:     subvolumes test-dht
> 51: end-volume
> 52: 
> 53: volume test-read-ahead
> 54:     type performance/read-ahead
> 55:     subvolumes test-write-behind
> 56: end-volume
> 57: 
> 58: volume test-io-cache
> 59:     type performance/io-cache
> 60:     subvolumes test-read-ahead
> 61: end-volume
> 62: 
> 63: volume test-quick-read
> 64:     type performance/quick-read
> 65:     subvolumes test-io-cache
> 66: end-volume
> 67: 
> 68: volume test-open-behind
> 69:     type performance/open-behind
> 70:     subvolumes test-quick-read
> 71: end-volume
> 72: 
> 73: volume test-md-cache
> 74:     type performance/md-cache
> 75:     subvolumes test-open-behind
> 76: end-volume
> 77: 
> 78: volume test
> 79:     type debug/io-stats
> 80:     option latency-measurement off
> 81:     option count-fop-hits off
> 82:     subvolumes test-md-cache
> 83: end-volume
> 84: 
>+------------------------------------------------------------------------------+
>[2014-08-12 21:04:23.255419] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test-client-0: changing port to 49152 (from 0)
>[2014-08-12 21:04:23.258052] E [client-handshake.c:1760:client_query_portmap_cbk] 1-test-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-08-12 21:04:23.258155] I [client.c:2229:client_rpc_notify] 1-test-client-2: disconnected from 192.168.30.107:24007. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-08-12 21:04:23.258173] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test-client-1: changing port to 49152 (from 0)
>[2014-08-12 21:04:23.258239] E [client-handshake.c:1760:client_query_portmap_cbk] 1-test-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-08-12 21:04:23.263012] I [client.c:2229:client_rpc_notify] 1-test-client-3: disconnected from 192.168.30.108:24007. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-08-12 21:04:23.263026] E [afr-common.c:4168:afr_notify] 1-test-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-08-12 21:04:23.266198] I [client-handshake.c:1677:select_server_supported_programs] 1-test-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:23.266428] I [client-handshake.c:1677:select_server_supported_programs] 1-test-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:23.266531] I [client-handshake.c:1462:client_setvolume_cbk] 1-test-client-0: Connected to 192.168.30.107:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 21:04:23.266542] I [client-handshake.c:1474:client_setvolume_cbk] 1-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:23.266611] I [afr-common.c:4131:afr_notify] 1-test-replicate-0: Subvolume 'test-client-0' came back up; going online.
>[2014-08-12 21:04:23.266728] I [client-handshake.c:1462:client_setvolume_cbk] 1-test-client-1: Connected to 192.168.30.108:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 21:04:23.266739] I [client-handshake.c:1474:client_setvolume_cbk] 1-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:23.269708] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 1
>[2014-08-12 21:04:23.269736] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test-client-1: Server lk version = 1
>[2014-08-12 21:04:23.269747] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test-client-0: Server lk version = 1
>[2014-08-12 21:04:23.335548] I [fuse-bridge.c:4189:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x15e72cc) to newfd (0x16b320c) (inode-gfid:e5e936cb-4ad9-4e56-8ace-980b74ce19af)(old-subvolume:test-0 new-subvolume:test-1)
>[2014-08-12 21:04:23.335604] I [client.c:2303:notify] 0-test-client-0: current graph is no longer active, destroying rpc_client 
>[2014-08-12 21:04:23.335653] I [client.c:2303:notify] 0-test-client-1: current graph is no longer active, destroying rpc_client 
>[2014-08-12 21:04:23.335889] I [client.c:2229:client_rpc_notify] 0-test-client-0: disconnected from 192.168.30.107:49152. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-08-12 21:04:23.335932] I [client.c:2229:client_rpc_notify] 0-test-client-1: disconnected from 192.168.30.108:49152. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-08-12 21:04:23.335941] E [afr-common.c:4168:afr_notify] 0-test-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-08-12 21:04:25.263497] W [defaults.c:1381:default_release] (-->/usr/lib64/glusterfs/3.5.2/xlator/protocol/client.so(client3_3_finodelk_cbk+0x153) [0x7f5600da68f3] (-->/usr/lib64/glusterfs/3.5.2/xlator/protocol/client.so(client_local_wipe+0x28) [0x7f5600d9ab88] (-->/usr/lib64/libglusterfs.so.0(fd_unref+0x10e) [0x7f56059f5fbe]))) 0-fuse: xlator does not implement release_cbk
>[2014-08-12 21:04:25.274366] I [afr-common.c:4299:afr_local_init] 1-test-replicate-1: no subvolumes up
>[2014-08-12 21:04:25.274390] W [dht-diskusage.c:45:dht_du_info_cbk] 1-test-dht: failed to get disk info from test-replicate-1
>[2014-08-12 21:04:25.275285] I [afr-common.c:4299:afr_local_init] 1-test-replicate-1: no subvolumes up
>[2014-08-12 21:04:26.789919] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test-client-2: changing port to 49153 (from 0)
>[2014-08-12 21:04:26.795465] I [client-handshake.c:1677:select_server_supported_programs] 1-test-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:26.795747] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 1-test-client-3: changing port to 49153 (from 0)
>[2014-08-12 21:04:26.798329] I [client-handshake.c:1462:client_setvolume_cbk] 1-test-client-2: Connected to 192.168.30.107:49153, attached to remote volume '/bricks/brick2/brick'.
>[2014-08-12 21:04:26.798341] I [client-handshake.c:1474:client_setvolume_cbk] 1-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:26.798462] I [afr-common.c:4131:afr_notify] 1-test-replicate-1: Subvolume 'test-client-2' came back up; going online.
>[2014-08-12 21:04:26.798593] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test-client-2: Server lk version = 1
>[2014-08-12 21:04:26.798711] I [client-handshake.c:1677:select_server_supported_programs] 1-test-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:26.802172] I [client-handshake.c:1462:client_setvolume_cbk] 1-test-client-3: Connected to 192.168.30.108:49153, attached to remote volume '/bricks/brick2/brick'.
>[2014-08-12 21:04:26.802184] I [client-handshake.c:1474:client_setvolume_cbk] 1-test-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:26.802378] I [client-handshake.c:450:client_set_lk_version_cbk] 1-test-client-3: Server lk version = 1
>[2014-08-12 21:04:30.216748] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:30.216775] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:32.685662] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:32.685686] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:35.130443] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:35.130467] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:37.576254] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:37.576281] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.034290] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.034314] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.037510] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.037522] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.046247] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.046262] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.054673] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.054688] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.063094] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.063108] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.071334] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.071348] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.079758] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.079773] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.087923] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.087938] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.096348] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.096362] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.104941] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.104954] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.113231] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.113245] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.121508] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.121522] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.130456] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.130470] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.138609] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.138624] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.146921] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.146935] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.154890] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.154904] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.162869] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.162882] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.170836] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.170849] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.178772] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.178786] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.186960] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.186973] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.195313] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.195337] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.203556] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.203569] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.211950] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.211962] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.220952] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.220965] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.229210] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.229222] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.237367] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.237379] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.245510] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.245522] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.253964] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.253976] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.262371] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.262383] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.270526] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.270538] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.278780] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.278793] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.286970] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.286983] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.295227] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.295239] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.303160] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.303173] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.311856] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.311868] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.319751] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.319764] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.327381] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.327394] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.335284] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.335296] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.343179] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.343192] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.351115] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.351165] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.359225] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.359237] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.362432] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-test-client-2: remote operation failed: Permission denied. Path: /14
>[2014-08-12 21:04:40.362532] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-test-client-3: remote operation failed: Permission denied. Path: /14
>[2014-08-12 21:04:40.364074] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:40.364085] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:40.365039] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-test-client-2: remote operation failed: Stale file handle. Path: /14 (ff6dee96-1569-4589-a6b4-5b3abe92d026)
>[2014-08-12 21:04:40.365215] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-test-client-3: remote operation failed: Stale file handle. Path: /14 (ff6dee96-1569-4589-a6b4-5b3abe92d026)
>[2014-08-12 21:04:40.367817] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:40.367954] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:40.368192] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:40.368536] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:40.368548] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:40.368554] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:04:42.804349] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:42.804374] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:42.808070] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:42.813595] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:42.813976] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:42.814400] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:42.814412] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:42.814417] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:04:45.256454] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:45.256478] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:45.260275] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:45.260369] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:45.260707] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:45.261150] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:45.261161] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:45.261167] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:04:47.699766] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:47.699789] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:47.703337] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:47.703495] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:47.703823] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:47.704270] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:47.704282] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:47.704288] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:04:50.142598] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:50.142623] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:50.146398] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:50.146532] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:50.146895] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:50.147365] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:50.147377] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:50.147383] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:04:52.598947] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.598970] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.602269] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.602280] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.611287] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.611299] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.619817] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.619829] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.628295] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.628306] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.636809] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.636822] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.645244] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.645256] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.653370] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.653381] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.661766] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.661778] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.669824] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.669848] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.678246] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.678258] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.686573] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.686584] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.695495] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.695507] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.703891] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.703903] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.712327] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.712339] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.720806] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.720818] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.729338] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.729350] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.737748] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.737759] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.745949] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.745961] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.754271] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.754283] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.762594] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.762606] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.770924] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.770936] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.779203] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.779215] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.787948] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.787960] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.796209] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.796222] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.804538] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.804550] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.813006] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.813018] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.821344] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.821357] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.829791] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.829815] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.838098] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.838110] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.846477] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.846489] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.854973] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.854985] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.863270] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.863282] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.871723] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.871735] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.880495] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.880508] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.889018] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.889054] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.897748] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.897760] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.905797] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.905809] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.913808] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.913821] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.921807] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.921819] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.929986] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.929998] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.938314] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.938326] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.946832] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.946845] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.955276] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.955289] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.963535] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.963547] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.972598] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.972610] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.980923] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.980935] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.989121] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.989170] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:52.997505] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:52.997518] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:53.006056] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:53.006069] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:53.014385] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:53.014397] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:53.022376] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:53.022387] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:53.030470] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:53.030482] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:53.033558] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-test-client-2: remote operation failed: Permission denied. Path: /19
>[2014-08-12 21:04:53.033661] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-test-client-3: remote operation failed: Permission denied. Path: /19
>[2014-08-12 21:04:53.035084] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:53.035095] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:53.036109] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-test-client-2: remote operation failed: Stale file handle. Path: /19 (a5c50e9c-07da-4533-8003-9c0ff96c7919)
>[2014-08-12 21:04:53.036130] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-test-client-3: remote operation failed: Stale file handle. Path: /19 (a5c50e9c-07da-4533-8003-9c0ff96c7919)
>[2014-08-12 21:04:53.038932] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:53.039001] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:53.039245] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:53.039540] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:53.039552] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:53.039557] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:04:55.491895] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:55.491921] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:55.495657] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:55.495749] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:55.496030] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:55.496427] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:55.496443] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:55.496451] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:04:58.015086] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:04:58.015121] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:04:58.018816] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:58.018908] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:58.019161] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:04:58.019505] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:04:58.019517] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:04:58.019522] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:00.476057] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:00.476082] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:00.479840] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:00.479927] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:00.480214] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:00.480663] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:00.480675] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:00.480681] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:02.969695] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:02.969724] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:02.972902] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:02.972914] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:02.981407] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:02.981420] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:02.990007] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:02.990019] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:02.998238] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:02.998251] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:03.001573] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-test-client-2: remote operation failed: Permission denied. Path: /20
>[2014-08-12 21:05:03.001699] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 1-test-client-3: remote operation failed: Permission denied. Path: /20
>[2014-08-12 21:05:03.003114] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:03.003126] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:03.004184] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-test-client-2: remote operation failed: Stale file handle. Path: /20 (f7f6d19d-9191-44fb-8bf5-f76d7780eda9)
>[2014-08-12 21:05:03.004202] W [client-rpc-fops.c:2690:client3_3_opendir_cbk] 1-test-client-3: remote operation failed: Stale file handle. Path: /20 (f7f6d19d-9191-44fb-8bf5-f76d7780eda9)
>[2014-08-12 21:05:03.007102] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:03.007184] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:03.007448] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:03.007846] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:03.007857] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:03.007863] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:05.512594] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:05.512619] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:05.516294] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:05.516424] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:05.516752] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:05.517266] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:05.517278] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:05.517284] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:07.968322] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:07.968345] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:07.972021] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:07.972173] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:07.973402] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:07.973825] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:07.973837] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:07.973843] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:10.442867] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:10.442891] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:10.446658] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:10.446699] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:10.447082] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:10.447495] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:10.447507] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:10.447513] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:12.890874] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:12.890898] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:12.894622] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:12.894786] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:12.895155] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:12.895582] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:12.895593] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:12.895599] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:15.338535] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:15.338562] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:15.342410] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:15.342452] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:15.342702] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:15.343107] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:15.343119] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:15.343129] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:17.880922] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:17.880946] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:17.884475] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:17.884571] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:17.884856] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:17.885240] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:17.885251] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:17.885257] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.
>[2014-08-12 21:05:20.375505] I [dht-layout.c:727:dht_layout_dir_mismatch] 1-test-dht: / - disk layout missing
>[2014-08-12 21:05:20.375532] I [dht-common.c:635:dht_revalidate_cbk] 1-test-dht: mismatching layouts for /
>[2014-08-12 21:05:20.379181] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:20.381205] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:20.381530] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-2: remote operation failed: Stale file handle
>[2014-08-12 21:05:20.381973] W [client-rpc-fops.c:1632:client3_3_entrylk_cbk] 1-test-client-3: remote operation failed: Stale file handle
>[2014-08-12 21:05:20.381985] I [afr-lk-common.c:1092:afr_lock_blocking] 1-test-replicate-1: unable to lock on even one child
>[2014-08-12 21:05:20.381991] I [afr-transaction.c:1295:afr_post_blocking_entrylk_cbk] 1-test-replicate-1: Blocking entrylks failed.

Actions: View

Attachments on bug 1129486: 926217 | 926218 | 926219