Bug 765566 (GLUSTER-3834)

Summary: Error with lost+found directory when using multiple disks
Product: [Community] GlusterFS Reporter: ahazelwood
Component: coreAssignee: Divya <divya>
Status: CLOSED EOL QA Contact:
Severity: low Docs Contact:
Priority: medium    
Version: mainlineCC: bugs, gluster-bugs, kbsingh, matorola, storage-doc
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-22 15:46:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: DP CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description ahazelwood 2011-11-30 14:34:04 UTC 
This is a new install using gluster 3.2.5 on Ubuntu 11.10 (x64).  I specified several new disks on two servers that were formatted using mkfs.ext4 as part of a new Distributed Replicated volume with a Replica of 2.  Gluster is unable to get past differences in the lost+found directories created as part of the process and hangs within 2 minutes.  

Ideally, this folder should be ignored and shouldn't come into play as it is system created.  

I've attached content from /var/log/glusterfs/mnt-test.log:

[2011-11-29 18:37:53.177299] I [glusterfsd.c:1493:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.2.5
[2011-11-29 18:37:53.200398] W [write-behind.c:3023:init] 0-test-write-behind: disabling write-behind for first 0 bytes
[2011-11-29 18:37:53.217233] I [client.c:1935:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.218711] I [client.c:1935:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.219520] I [client.c:1935:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.220761] I [client.c:1935:notify] 0-test-client-3: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.222412] I [client.c:1935:notify] 0-test-client-4: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.223680] I [client.c:1935:notify] 0-test-client-5: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.226797] I [client.c:1935:notify] 0-test-client-6: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.227920] I [client.c:1935:notify] 0-test-client-7: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.229979] I [client.c:1935:notify] 0-test-client-8: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.232446] I [client.c:1935:notify] 0-test-client-9: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.233399] I [client.c:1935:notify] 0-test-client-10: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.235243] I [client.c:1935:notify] 0-test-client-11: parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
  1: volume test-client-0
  2:     type protocol/client
  3:     option remote-host dsdev1
  4:     option remote-subvolume /mnt/sdb
  5:     option transport-type tcp
  6: end-volume
  7:
  8: volume test-client-1
  9:     type protocol/client
 10:     option remote-host dsdev2
 11:     option remote-subvolume /mnt/sdb
 12:     option transport-type tcp
 13: end-volume
 14:
 15: volume test-client-2
 16:     type protocol/client
 17:     option remote-host dsdev1
 18:     option remote-subvolume /mnt/sdc
 19:     option transport-type tcp
 20: end-volume
 21:
 22: volume test-client-3
 23:     type protocol/client
 24:     option remote-host dsdev2
 25:     option remote-subvolume /mnt/sdc
 26:     option transport-type tcp
 27: end-volume
 28:
 29: volume test-client-4
 30:     type protocol/client
 31:     option remote-host dsdev1
 32:     option remote-subvolume /mnt/sdd
 33:     option transport-type tcp
 34: end-volume
 35:
 36: volume test-client-5
 37:     type protocol/client
 38:     option remote-host dsdev2
 39:     option remote-subvolume /mnt/sdd
 40:     option transport-type tcp
 41: end-volume
 42:
 43: volume test-client-6
 44:     type protocol/client
 45:     option remote-host dsdev1
 46:     option remote-subvolume /mnt/sde
 47:     option transport-type tcp
 48: end-volume
 49:
 50: volume test-client-7
 51:     type protocol/client
 52:     option remote-host dsdev2
 53:     option remote-subvolume /mnt/sde
 54:     option transport-type tcp
 55: end-volume
 56:
 57: volume test-client-8
 58:     type protocol/client
 59:     option remote-host dsdev1
 60:     option remote-subvolume /mnt/sdf
 61:     option transport-type tcp
 62: end-volume
 63:
 64: volume test-client-9
 65:     type protocol/client
 66:     option remote-host dsdev2
 67:     option remote-subvolume /mnt/sdf
 68:     option transport-type tcp
 69: end-volume
 70:
 71: volume test-client-10
 72:     type protocol/client
 73:     option remote-host dsdev1
 74:     option remote-subvolume /mnt/sdg
 75:     option transport-type tcp
 76: end-volume
 77:
 78: volume test-client-11
 79:     type protocol/client
 80:     option remote-host dsdev2
 81:     option remote-subvolume /mnt/sdg
 82:     option transport-type tcp
 83: end-volume
 84:
 85: volume test-replicate-0
 86:     type cluster/replicate
 87:     subvolumes test-client-0 test-client-1
 88: end-volume
 89:
 90: volume test-replicate-1
 91:     type cluster/replicate
 92:     subvolumes test-client-2 test-client-3
 93: end-volume
 94:
 95: volume test-replicate-2
 96:     type cluster/replicate
 97:     subvolumes test-client-4 test-client-5
 98: end-volume
 99:
100: volume test-replicate-3
101:     type cluster/replicate
102:     subvolumes test-client-6 test-client-7
103: end-volume
104:
105: volume test-replicate-4
106:     type cluster/replicate
107:     subvolumes test-client-8 test-client-9
108: end-volume
109:
110: volume test-replicate-5
111:     type cluster/replicate
112:     subvolumes test-client-10 test-client-11
113: end-volume
114:
115: volume test-dht
116:     type cluster/distribute
117:     subvolumes test-replicate-0 test-replicate-1 test-replicate-2 test-replicate-3 test-replicate-4 test-replicate-5
118: end-volume
119:
120: volume test-write-behind
121:     type performance/write-behind
122:     subvolumes test-dht
123: end-volume
124:
125: volume test-read-ahead
126:     type performance/read-ahead
127:     subvolumes test-write-behind
128: end-volume
129:
130: volume test-io-cache
131:     type performance/io-cache
132:     subvolumes test-read-ahead
133: end-volume
134:
135: volume test-quick-read
136:     type performance/quick-read
137:     subvolumes test-io-cache
138: end-volume
139:
140: volume test-stat-prefetch
141:     type performance/stat-prefetch
142:     subvolumes test-quick-read
143: end-volume
144:
145: volume test
146:     type debug/io-stats
147:     option latency-measurement off
148:     option count-fop-hits off
149:     subvolumes test-stat-prefetch
150: end-volume

+------------------------------------------------------------------------------+
[2011-11-29 18:37:53.243531] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-0: changing port to 24010 (from 0)
[2011-11-29 18:37:53.244720] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-2: changing port to 24011 (from 0)
[2011-11-29 18:37:53.244772] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-4: changing port to 24012 (from 0)
[2011-11-29 18:37:53.244990] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-6: changing port to 24013 (from 0)
[2011-11-29 18:37:53.245036] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-8: changing port to 24014 (from 0)
[2011-11-29 18:37:53.245107] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-10: changing port to 24015 (from 0)
[2011-11-29 18:37:53.245455] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-1: changing port to 24010 (from 0)
[2011-11-29 18:37:53.245494] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-3: changing port to 24011 (from 0)
[2011-11-29 18:37:53.247282] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-5: changing port to 24012 (from 0)
[2011-11-29 18:37:53.247325] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-7: changing port to 24013 (from 0)
[2011-11-29 18:37:53.247356] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-9: changing port to 24014 (from 0)
[2011-11-29 18:37:53.251317] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-11: changing port to 24015 (from 0)
[2011-11-29 18:37:57.188998] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.190310] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-0: Connected to 192.168.3.1:24010, attached to remote volume '/mnt/sdb'.
[2011-11-29 18:37:57.190345] I [afr-common.c:3141:afr_notify] 0-test-replicate-0: Subvolume 'test-client-0' came back up; going online.
[2011-11-29 18:37:57.191217] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.191564] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-2: Connected to 192.168.3.1:24011, attached to remote volume '/mnt/sdc'.
[2011-11-29 18:37:57.192331] I [afr-common.c:3141:afr_notify] 0-test-replicate-1: Subvolume 'test-client-2' came back up; going online.
[2011-11-29 18:37:57.192419] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-4: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.192661] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-4: Connected to 192.168.3.1:24012, attached to remote volume '/mnt/sdd'.
[2011-11-29 18:37:57.192685] I [afr-common.c:3141:afr_notify] 0-test-replicate-2: Subvolume 'test-client-4' came back up; going online.
[2011-11-29 18:37:57.193568] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-6: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.194686] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-6: Connected to 192.168.3.1:24013, attached to remote volume '/mnt/sde'.
[2011-11-29 18:37:57.194711] I [afr-common.c:3141:afr_notify] 0-test-replicate-3: Subvolume 'test-client-6' came back up; going online.
[2011-11-29 18:37:57.194738] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-8: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.196050] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-8: Connected to 192.168.3.1:24014, attached to remote volume '/mnt/sdf'.
[2011-11-29 18:37:57.196076] I [afr-common.c:3141:afr_notify] 0-test-replicate-4: Subvolume 'test-client-8' came back up; going online.
[2011-11-29 18:37:57.196105] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-10: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.196993] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-10: Connected to 192.168.3.1:24015, attached to remote volume '/mnt/sdg'.
[2011-11-29 18:37:57.197018] I [afr-common.c:3141:afr_notify] 0-test-replicate-5: Subvolume 'test-client-10' came back up; going online.
[2011-11-29 18:37:57.199627] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.200843] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-3: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.200908] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-5: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.202726] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-3: Connected to 192.168.3.2:24011, attached to remote volume '/mnt/sdc'.
[2011-11-29 18:37:57.203950] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-7: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.204077] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-1: Connected to 192.168.3.2:24010, attached to remote volume '/mnt/sdb'.
[2011-11-29 18:37:57.205230] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-9: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.205370] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-7: Connected to 192.168.3.2:24013, attached to remote volume '/mnt/sde'.
[2011-11-29 18:37:57.206095] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-5: Connected to 192.168.3.2:24012, attached to remote volume '/mnt/sdd'.
[2011-11-29 18:37:57.206860] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-9: Connected to 192.168.3.2:24014, attached to remote volume '/mnt/sdf'.
[2011-11-29 18:37:57.207677] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-11: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.209131] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-11: Connected to 192.168.3.2:24015, attached to remote volume '/mnt/sdg'.
[2011-11-29 18:37:57.213300] I [fuse-bridge.c:3339:fuse_graph_setup] 0-fuse: switched to graph 0
[2011-11-29 18:37:57.213538] I [fuse-bridge.c:2927:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16
[2011-11-29 18:37:57.214785] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-0: added root inode
[2011-11-29 18:37:57.223771] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-1: added root inode
[2011-11-29 18:37:57.223815] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-2: added root inode
[2011-11-29 18:37:57.223847] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-3: added root inode
[2011-11-29 18:37:57.224207] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-4: added root inode
[2011-11-29 18:37:57.224299] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-5: added root inode
[2011-11-29 18:37:59.706090] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:37:59.706222] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:37:59.707070] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.709197] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.710551] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.712251] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.712277] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:37:59.712295] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:37:59.713327] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.714566] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.715055] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.718001] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.718032] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:37:59.718986] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:15.326930] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:15.326998] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:15.356263] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:38:15.357185] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:38:15.360324] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.361557] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.363391] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.368037] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.368066] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:15.368081] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:15.373134] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.373666] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.376606] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.378826] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.378856] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:15.378871] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:56.418045] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:56.418207] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:56.424212] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:38:56.425218] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:38:56.428004] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.428483] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.430055] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.434384] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.434478] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:56.434581] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:56.438663] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.441256] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.444564] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.446613] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.446707] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:56.446814] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:56.449230] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.450378] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.451742] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.452876] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.631225] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.633071] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.638323] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.639472] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.651353] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.653039] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.664758] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.668635] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.681062] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.682423] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.743580] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.744993] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.756600] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.757805] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.765987] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.767405] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.983472] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:33.983620] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:33.988224] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:39:33.988741] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:39:33.990216] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:33.990312] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.991275] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:33.993757] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.994096] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:33.994160] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:39:33.998003] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:33.998334] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.999673] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:34.772] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:34.1130] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:34.1507] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:39:34.4248] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:34.5176] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:34.6302] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:34.7090] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.367727] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:43.368253] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:43.373007] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:39:43.375917] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:39:43.382030] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.382155] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.382461] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.384980] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.385099] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:43.385180] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:39:43.390265] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.393514] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.394545] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.397245] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.397339] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:43.397417] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:33.443714] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:33.444153] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:33.448884] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:40:33.449820] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:40:33.451464] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.453891] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.454991] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.457787] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.458849] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:40:33.458935] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:33.461319] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.461494] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.463903] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.466504] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.466599] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:40:33.467119] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:36.214909] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:36.215383] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:36.223477] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:40:36.224517] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:40:36.228495] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.229136] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.229438] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.240941] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.241168] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:40:36.242572] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:36.255641] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.255771] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.260899] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.265115] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.266560] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
Comment 1 Amar Tumballi 2011-12-01 07:32:56 UTC 
We acknowledge that this may be an issue now. But normally we recommend to use a directory inside your mount as export. (if /dev/sdb1 is mounted in /gfs_export/, don't use /gfs_export/ as brick dir, instead use /gfs_export/<VOLNAME> or something like that as export dir)

With that, in glusterfs exported directory, you won't have 'lost+found' directory itself.

Thanks for having the bug report, we will work on fix this glitch, but as there is a work around, we treat it as an enhancement.
Comment 2 ahazelwood 2011-12-01 11:39:10 UTC 
Thanks for looking at this.  I wasn't sure how it should be marked, so enhancement makes sense. 

Please at least add this to the installation instructions as potentially a note as it would have saved me some time trying to track down.

Thanks.
Comment 3 Amar Tumballi 2012-04-17 18:43:55 UTC 
Not working on fixing this, should surely need to document this behavior.
Comment 4 Anatoly Pugachev 2014-12-11 09:57:15 UTC 
Can you please advice how to move from using block device mount as a brick to subdirectory in glusterfs. For example, move all data from /exports/brick1/ (/dev/sdb1 mounted) to /exports/brick1/datadir/ . thanks.
Comment 5 Kaleb KEITHLEY 2015-10-22 15:46:38 UTC 
because of the large number of bugs filed against mainline version\ is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.