This is a new install using gluster 3.2.5 on Ubuntu 11.10 (x64). I specified several new disks on two servers that were formatted using mkfs.ext4 as part of a new Distributed Replicated volume with a Replica of 2. Gluster is unable to get past differences in the lost+found directories created as part of the process and hangs within 2 minutes. Ideally, this folder should be ignored and shouldn't come into play as it is system created. I've attached content from /var/log/glusterfs/mnt-test.log: [2011-11-29 18:37:53.177299] I [glusterfsd.c:1493:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.2.5 [2011-11-29 18:37:53.200398] W [write-behind.c:3023:init] 0-test-write-behind: disabling write-behind for first 0 bytes [2011-11-29 18:37:53.217233] I [client.c:1935:notify] 0-test-client-0: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.218711] I [client.c:1935:notify] 0-test-client-1: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.219520] I [client.c:1935:notify] 0-test-client-2: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.220761] I [client.c:1935:notify] 0-test-client-3: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.222412] I [client.c:1935:notify] 0-test-client-4: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.223680] I [client.c:1935:notify] 0-test-client-5: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.226797] I [client.c:1935:notify] 0-test-client-6: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.227920] I [client.c:1935:notify] 0-test-client-7: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.229979] I [client.c:1935:notify] 0-test-client-8: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.232446] I [client.c:1935:notify] 0-test-client-9: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.233399] I [client.c:1935:notify] 0-test-client-10: parent translators are ready, attempting connect on transport [2011-11-29 18:37:53.235243] I [client.c:1935:notify] 0-test-client-11: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume test-client-0 2: type protocol/client 3: option remote-host dsdev1 4: option remote-subvolume /mnt/sdb 5: option transport-type tcp 6: end-volume 7: 8: volume test-client-1 9: type protocol/client 10: option remote-host dsdev2 11: option remote-subvolume /mnt/sdb 12: option transport-type tcp 13: end-volume 14: 15: volume test-client-2 16: type protocol/client 17: option remote-host dsdev1 18: option remote-subvolume /mnt/sdc 19: option transport-type tcp 20: end-volume 21: 22: volume test-client-3 23: type protocol/client 24: option remote-host dsdev2 25: option remote-subvolume /mnt/sdc 26: option transport-type tcp 27: end-volume 28: 29: volume test-client-4 30: type protocol/client 31: option remote-host dsdev1 32: option remote-subvolume /mnt/sdd 33: option transport-type tcp 34: end-volume 35: 36: volume test-client-5 37: type protocol/client 38: option remote-host dsdev2 39: option remote-subvolume /mnt/sdd 40: option transport-type tcp 41: end-volume 42: 43: volume test-client-6 44: type protocol/client 45: option remote-host dsdev1 46: option remote-subvolume /mnt/sde 47: option transport-type tcp 48: end-volume 49: 50: volume test-client-7 51: type protocol/client 52: option remote-host dsdev2 53: option remote-subvolume /mnt/sde 54: option transport-type tcp 55: end-volume 56: 57: volume test-client-8 58: type protocol/client 59: option remote-host dsdev1 60: option remote-subvolume /mnt/sdf 61: option transport-type tcp 62: end-volume 63: 64: volume test-client-9 65: type protocol/client 66: option remote-host dsdev2 67: option remote-subvolume /mnt/sdf 68: option transport-type tcp 69: end-volume 70: 71: volume test-client-10 72: type protocol/client 73: option remote-host dsdev1 74: option remote-subvolume /mnt/sdg 75: option transport-type tcp 76: end-volume 77: 78: volume test-client-11 79: type protocol/client 80: option remote-host dsdev2 81: option remote-subvolume /mnt/sdg 82: option transport-type tcp 83: end-volume 84: 85: volume test-replicate-0 86: type cluster/replicate 87: subvolumes test-client-0 test-client-1 88: end-volume 89: 90: volume test-replicate-1 91: type cluster/replicate 92: subvolumes test-client-2 test-client-3 93: end-volume 94: 95: volume test-replicate-2 96: type cluster/replicate 97: subvolumes test-client-4 test-client-5 98: end-volume 99: 100: volume test-replicate-3 101: type cluster/replicate 102: subvolumes test-client-6 test-client-7 103: end-volume 104: 105: volume test-replicate-4 106: type cluster/replicate 107: subvolumes test-client-8 test-client-9 108: end-volume 109: 110: volume test-replicate-5 111: type cluster/replicate 112: subvolumes test-client-10 test-client-11 113: end-volume 114: 115: volume test-dht 116: type cluster/distribute 117: subvolumes test-replicate-0 test-replicate-1 test-replicate-2 test-replicate-3 test-replicate-4 test-replicate-5 118: end-volume 119: 120: volume test-write-behind 121: type performance/write-behind 122: subvolumes test-dht 123: end-volume 124: 125: volume test-read-ahead 126: type performance/read-ahead 127: subvolumes test-write-behind 128: end-volume 129: 130: volume test-io-cache 131: type performance/io-cache 132: subvolumes test-read-ahead 133: end-volume 134: 135: volume test-quick-read 136: type performance/quick-read 137: subvolumes test-io-cache 138: end-volume 139: 140: volume test-stat-prefetch 141: type performance/stat-prefetch 142: subvolumes test-quick-read 143: end-volume 144: 145: volume test 146: type debug/io-stats 147: option latency-measurement off 148: option count-fop-hits off 149: subvolumes test-stat-prefetch 150: end-volume +------------------------------------------------------------------------------+ [2011-11-29 18:37:53.243531] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-0: changing port to 24010 (from 0) [2011-11-29 18:37:53.244720] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-2: changing port to 24011 (from 0) [2011-11-29 18:37:53.244772] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-4: changing port to 24012 (from 0) [2011-11-29 18:37:53.244990] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-6: changing port to 24013 (from 0) [2011-11-29 18:37:53.245036] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-8: changing port to 24014 (from 0) [2011-11-29 18:37:53.245107] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-10: changing port to 24015 (from 0) [2011-11-29 18:37:53.245455] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-1: changing port to 24010 (from 0) [2011-11-29 18:37:53.245494] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-3: changing port to 24011 (from 0) [2011-11-29 18:37:53.247282] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-5: changing port to 24012 (from 0) [2011-11-29 18:37:53.247325] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-7: changing port to 24013 (from 0) [2011-11-29 18:37:53.247356] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-9: changing port to 24014 (from 0) [2011-11-29 18:37:53.251317] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-11: changing port to 24015 (from 0) [2011-11-29 18:37:57.188998] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.190310] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-0: Connected to 192.168.3.1:24010, attached to remote volume '/mnt/sdb'. [2011-11-29 18:37:57.190345] I [afr-common.c:3141:afr_notify] 0-test-replicate-0: Subvolume 'test-client-0' came back up; going online. [2011-11-29 18:37:57.191217] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.191564] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-2: Connected to 192.168.3.1:24011, attached to remote volume '/mnt/sdc'. [2011-11-29 18:37:57.192331] I [afr-common.c:3141:afr_notify] 0-test-replicate-1: Subvolume 'test-client-2' came back up; going online. [2011-11-29 18:37:57.192419] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-4: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.192661] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-4: Connected to 192.168.3.1:24012, attached to remote volume '/mnt/sdd'. [2011-11-29 18:37:57.192685] I [afr-common.c:3141:afr_notify] 0-test-replicate-2: Subvolume 'test-client-4' came back up; going online. [2011-11-29 18:37:57.193568] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-6: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.194686] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-6: Connected to 192.168.3.1:24013, attached to remote volume '/mnt/sde'. [2011-11-29 18:37:57.194711] I [afr-common.c:3141:afr_notify] 0-test-replicate-3: Subvolume 'test-client-6' came back up; going online. [2011-11-29 18:37:57.194738] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-8: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.196050] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-8: Connected to 192.168.3.1:24014, attached to remote volume '/mnt/sdf'. [2011-11-29 18:37:57.196076] I [afr-common.c:3141:afr_notify] 0-test-replicate-4: Subvolume 'test-client-8' came back up; going online. [2011-11-29 18:37:57.196105] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-10: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.196993] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-10: Connected to 192.168.3.1:24015, attached to remote volume '/mnt/sdg'. [2011-11-29 18:37:57.197018] I [afr-common.c:3141:afr_notify] 0-test-replicate-5: Subvolume 'test-client-10' came back up; going online. [2011-11-29 18:37:57.199627] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.200843] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-3: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.200908] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-5: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.202726] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-3: Connected to 192.168.3.2:24011, attached to remote volume '/mnt/sdc'. [2011-11-29 18:37:57.203950] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-7: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.204077] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-1: Connected to 192.168.3.2:24010, attached to remote volume '/mnt/sdb'. [2011-11-29 18:37:57.205230] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-9: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.205370] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-7: Connected to 192.168.3.2:24013, attached to remote volume '/mnt/sde'. [2011-11-29 18:37:57.206095] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-5: Connected to 192.168.3.2:24012, attached to remote volume '/mnt/sdd'. [2011-11-29 18:37:57.206860] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-9: Connected to 192.168.3.2:24014, attached to remote volume '/mnt/sdf'. [2011-11-29 18:37:57.207677] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-11: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2011-11-29 18:37:57.209131] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-11: Connected to 192.168.3.2:24015, attached to remote volume '/mnt/sdg'. [2011-11-29 18:37:57.213300] I [fuse-bridge.c:3339:fuse_graph_setup] 0-fuse: switched to graph 0 [2011-11-29 18:37:57.213538] I [fuse-bridge.c:2927:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16 [2011-11-29 18:37:57.214785] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-0: added root inode [2011-11-29 18:37:57.223771] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-1: added root inode [2011-11-29 18:37:57.223815] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-2: added root inode [2011-11-29 18:37:57.223847] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-3: added root inode [2011-11-29 18:37:57.224207] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-4: added root inode [2011-11-29 18:37:57.224299] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-5: added root inode [2011-11-29 18:37:59.706090] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0 [2011-11-29 18:37:59.706222] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0 [2011-11-29 18:37:59.707070] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:37:59.709197] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:37:59.710551] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:37:59.712251] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:37:59.712277] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:37:59.712295] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:37:59.713327] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:37:59.714566] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:37:59.715055] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:37:59.718001] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:37:59.718032] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:37:59.718986] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:38:15.326930] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle [2011-11-29 18:38:15.326998] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle [2011-11-29 18:38:15.356263] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0 [2011-11-29 18:38:15.357185] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0 [2011-11-29 18:38:15.360324] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:15.361557] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:15.363391] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:15.368037] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:15.368066] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:38:15.368081] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:38:15.373134] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:15.373666] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:15.376606] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:15.378826] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:15.378856] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:38:15.378871] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:38:56.418045] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle [2011-11-29 18:38:56.418207] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle [2011-11-29 18:38:56.424212] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0 [2011-11-29 18:38:56.425218] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0 [2011-11-29 18:38:56.428004] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.428483] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.430055] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.434384] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.434478] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:38:56.434581] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:38:56.438663] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.441256] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.444564] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.446613] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.446707] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:38:56.446814] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:38:56.449230] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.450378] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.451742] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.452876] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.631225] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.633071] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.638323] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.639472] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.651353] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.653039] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.664758] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.668635] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.681062] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.682423] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.743580] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.744993] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.756600] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.757805] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:38:56.765987] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:38:56.767405] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:33.983472] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle [2011-11-29 18:39:33.983620] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle [2011-11-29 18:39:33.988224] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0 [2011-11-29 18:39:33.988741] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0 [2011-11-29 18:39:33.990216] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:33.990312] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:33.991275] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:33.993757] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:33.994096] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:39:33.994160] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:39:33.998003] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:33.998334] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:33.999673] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:34.772] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:34.1130] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:39:34.1507] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:39:34.4248] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:34.5176] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:34.6302] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:34.7090] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:43.367727] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle [2011-11-29 18:39:43.368253] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle [2011-11-29 18:39:43.373007] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0 [2011-11-29 18:39:43.375917] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0 [2011-11-29 18:39:43.382030] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:43.382155] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:43.382461] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:43.384980] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:43.385099] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:39:43.385180] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:39:43.390265] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:43.393514] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:43.394545] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:39:43.397245] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:39:43.397339] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:39:43.397417] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:40:33.443714] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle [2011-11-29 18:40:33.444153] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle [2011-11-29 18:40:33.448884] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0 [2011-11-29 18:40:33.449820] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0 [2011-11-29 18:40:33.451464] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:33.453891] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:33.454991] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:33.457787] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:33.458849] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:40:33.458935] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:40:33.461319] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:33.461494] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:33.463903] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:33.466504] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:33.466599] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:40:33.467119] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:40:36.214909] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle [2011-11-29 18:40:36.215383] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle [2011-11-29 18:40:36.223477] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0 [2011-11-29 18:40:36.224517] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0 [2011-11-29 18:40:36.228495] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:36.229136] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:36.229438] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:36.240941] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:36.241168] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child [2011-11-29 18:40:36.242572] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed. [2011-11-29 18:40:36.255641] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:36.255771] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:36.260899] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory [2011-11-29 18:40:36.265115] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory [2011-11-29 18:40:36.266560] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
We acknowledge that this may be an issue now. But normally we recommend to use a directory inside your mount as export. (if /dev/sdb1 is mounted in /gfs_export/, don't use /gfs_export/ as brick dir, instead use /gfs_export/<VOLNAME> or something like that as export dir) With that, in glusterfs exported directory, you won't have 'lost+found' directory itself. Thanks for having the bug report, we will work on fix this glitch, but as there is a work around, we treat it as an enhancement.
Thanks for looking at this. I wasn't sure how it should be marked, so enhancement makes sense. Please at least add this to the installation instructions as potentially a note as it would have saved me some time trying to track down. Thanks.
Not working on fixing this, should surely need to document this behavior.
Can you please advice how to move from using block device mount as a brick to subdirectory in glusterfs. For example, move all data from /exports/brick1/ (/dev/sdb1 mounted) to /exports/brick1/datadir/ . thanks.
because of the large number of bugs filed against mainline version\ is ambiguous and about to be removed as a choice. If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.