Bug 765566 (GLUSTER-3834) - Error with lost+found directory when using multiple disks
Summary: Error with lost+found directory when using multiple disks
Keywords:
Status: CLOSED EOL
Alias: GLUSTER-3834
Product: GlusterFS
Classification: Community
Component: core
Version: mainline
Hardware: x86_64
OS: Linux
medium
low
Target Milestone: ---
Assignee: Divya
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-11-30 14:34 UTC by ahazelwood
Modified: 2015-10-22 15:46 UTC (History)
5 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-10-22 15:46:38 UTC
Regression: ---
Mount Type: ---
Documentation: DP
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description ahazelwood 2011-11-30 14:34:04 UTC
This is a new install using gluster 3.2.5 on Ubuntu 11.10 (x64).  I specified several new disks on two servers that were formatted using mkfs.ext4 as part of a new Distributed Replicated volume with a Replica of 2.  Gluster is unable to get past differences in the lost+found directories created as part of the process and hangs within 2 minutes.  

Ideally, this folder should be ignored and shouldn't come into play as it is system created.  

I've attached content from /var/log/glusterfs/mnt-test.log:

[2011-11-29 18:37:53.177299] I [glusterfsd.c:1493:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.2.5
[2011-11-29 18:37:53.200398] W [write-behind.c:3023:init] 0-test-write-behind: disabling write-behind for first 0 bytes
[2011-11-29 18:37:53.217233] I [client.c:1935:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.218711] I [client.c:1935:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.219520] I [client.c:1935:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.220761] I [client.c:1935:notify] 0-test-client-3: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.222412] I [client.c:1935:notify] 0-test-client-4: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.223680] I [client.c:1935:notify] 0-test-client-5: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.226797] I [client.c:1935:notify] 0-test-client-6: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.227920] I [client.c:1935:notify] 0-test-client-7: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.229979] I [client.c:1935:notify] 0-test-client-8: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.232446] I [client.c:1935:notify] 0-test-client-9: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.233399] I [client.c:1935:notify] 0-test-client-10: parent translators are ready, attempting connect on transport
[2011-11-29 18:37:53.235243] I [client.c:1935:notify] 0-test-client-11: parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
  1: volume test-client-0
  2:     type protocol/client
  3:     option remote-host dsdev1
  4:     option remote-subvolume /mnt/sdb
  5:     option transport-type tcp
  6: end-volume
  7:
  8: volume test-client-1
  9:     type protocol/client
 10:     option remote-host dsdev2
 11:     option remote-subvolume /mnt/sdb
 12:     option transport-type tcp
 13: end-volume
 14:
 15: volume test-client-2
 16:     type protocol/client
 17:     option remote-host dsdev1
 18:     option remote-subvolume /mnt/sdc
 19:     option transport-type tcp
 20: end-volume
 21:
 22: volume test-client-3
 23:     type protocol/client
 24:     option remote-host dsdev2
 25:     option remote-subvolume /mnt/sdc
 26:     option transport-type tcp
 27: end-volume
 28:
 29: volume test-client-4
 30:     type protocol/client
 31:     option remote-host dsdev1
 32:     option remote-subvolume /mnt/sdd
 33:     option transport-type tcp
 34: end-volume
 35:
 36: volume test-client-5
 37:     type protocol/client
 38:     option remote-host dsdev2
 39:     option remote-subvolume /mnt/sdd
 40:     option transport-type tcp
 41: end-volume
 42:
 43: volume test-client-6
 44:     type protocol/client
 45:     option remote-host dsdev1
 46:     option remote-subvolume /mnt/sde
 47:     option transport-type tcp
 48: end-volume
 49:
 50: volume test-client-7
 51:     type protocol/client
 52:     option remote-host dsdev2
 53:     option remote-subvolume /mnt/sde
 54:     option transport-type tcp
 55: end-volume
 56:
 57: volume test-client-8
 58:     type protocol/client
 59:     option remote-host dsdev1
 60:     option remote-subvolume /mnt/sdf
 61:     option transport-type tcp
 62: end-volume
 63:
 64: volume test-client-9
 65:     type protocol/client
 66:     option remote-host dsdev2
 67:     option remote-subvolume /mnt/sdf
 68:     option transport-type tcp
 69: end-volume
 70:
 71: volume test-client-10
 72:     type protocol/client
 73:     option remote-host dsdev1
 74:     option remote-subvolume /mnt/sdg
 75:     option transport-type tcp
 76: end-volume
 77:
 78: volume test-client-11
 79:     type protocol/client
 80:     option remote-host dsdev2
 81:     option remote-subvolume /mnt/sdg
 82:     option transport-type tcp
 83: end-volume
 84:
 85: volume test-replicate-0
 86:     type cluster/replicate
 87:     subvolumes test-client-0 test-client-1
 88: end-volume
 89:
 90: volume test-replicate-1
 91:     type cluster/replicate
 92:     subvolumes test-client-2 test-client-3
 93: end-volume
 94:
 95: volume test-replicate-2
 96:     type cluster/replicate
 97:     subvolumes test-client-4 test-client-5
 98: end-volume
 99:
100: volume test-replicate-3
101:     type cluster/replicate
102:     subvolumes test-client-6 test-client-7
103: end-volume
104:
105: volume test-replicate-4
106:     type cluster/replicate
107:     subvolumes test-client-8 test-client-9
108: end-volume
109:
110: volume test-replicate-5
111:     type cluster/replicate
112:     subvolumes test-client-10 test-client-11
113: end-volume
114:
115: volume test-dht
116:     type cluster/distribute
117:     subvolumes test-replicate-0 test-replicate-1 test-replicate-2 test-replicate-3 test-replicate-4 test-replicate-5
118: end-volume
119:
120: volume test-write-behind
121:     type performance/write-behind
122:     subvolumes test-dht
123: end-volume
124:
125: volume test-read-ahead
126:     type performance/read-ahead
127:     subvolumes test-write-behind
128: end-volume
129:
130: volume test-io-cache
131:     type performance/io-cache
132:     subvolumes test-read-ahead
133: end-volume
134:
135: volume test-quick-read
136:     type performance/quick-read
137:     subvolumes test-io-cache
138: end-volume
139:
140: volume test-stat-prefetch
141:     type performance/stat-prefetch
142:     subvolumes test-quick-read
143: end-volume
144:
145: volume test
146:     type debug/io-stats
147:     option latency-measurement off
148:     option count-fop-hits off
149:     subvolumes test-stat-prefetch
150: end-volume

+------------------------------------------------------------------------------+
[2011-11-29 18:37:53.243531] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-0: changing port to 24010 (from 0)
[2011-11-29 18:37:53.244720] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-2: changing port to 24011 (from 0)
[2011-11-29 18:37:53.244772] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-4: changing port to 24012 (from 0)
[2011-11-29 18:37:53.244990] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-6: changing port to 24013 (from 0)
[2011-11-29 18:37:53.245036] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-8: changing port to 24014 (from 0)
[2011-11-29 18:37:53.245107] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-10: changing port to 24015 (from 0)
[2011-11-29 18:37:53.245455] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-1: changing port to 24010 (from 0)
[2011-11-29 18:37:53.245494] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-3: changing port to 24011 (from 0)
[2011-11-29 18:37:53.247282] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-5: changing port to 24012 (from 0)
[2011-11-29 18:37:53.247325] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-7: changing port to 24013 (from 0)
[2011-11-29 18:37:53.247356] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-9: changing port to 24014 (from 0)
[2011-11-29 18:37:53.251317] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-test-client-11: changing port to 24015 (from 0)
[2011-11-29 18:37:57.188998] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.190310] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-0: Connected to 192.168.3.1:24010, attached to remote volume '/mnt/sdb'.
[2011-11-29 18:37:57.190345] I [afr-common.c:3141:afr_notify] 0-test-replicate-0: Subvolume 'test-client-0' came back up; going online.
[2011-11-29 18:37:57.191217] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.191564] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-2: Connected to 192.168.3.1:24011, attached to remote volume '/mnt/sdc'.
[2011-11-29 18:37:57.192331] I [afr-common.c:3141:afr_notify] 0-test-replicate-1: Subvolume 'test-client-2' came back up; going online.
[2011-11-29 18:37:57.192419] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-4: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.192661] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-4: Connected to 192.168.3.1:24012, attached to remote volume '/mnt/sdd'.
[2011-11-29 18:37:57.192685] I [afr-common.c:3141:afr_notify] 0-test-replicate-2: Subvolume 'test-client-4' came back up; going online.
[2011-11-29 18:37:57.193568] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-6: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.194686] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-6: Connected to 192.168.3.1:24013, attached to remote volume '/mnt/sde'.
[2011-11-29 18:37:57.194711] I [afr-common.c:3141:afr_notify] 0-test-replicate-3: Subvolume 'test-client-6' came back up; going online.
[2011-11-29 18:37:57.194738] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-8: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.196050] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-8: Connected to 192.168.3.1:24014, attached to remote volume '/mnt/sdf'.
[2011-11-29 18:37:57.196076] I [afr-common.c:3141:afr_notify] 0-test-replicate-4: Subvolume 'test-client-8' came back up; going online.
[2011-11-29 18:37:57.196105] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-10: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.196993] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-10: Connected to 192.168.3.1:24015, attached to remote volume '/mnt/sdg'.
[2011-11-29 18:37:57.197018] I [afr-common.c:3141:afr_notify] 0-test-replicate-5: Subvolume 'test-client-10' came back up; going online.
[2011-11-29 18:37:57.199627] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.200843] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-3: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.200908] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-5: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.202726] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-3: Connected to 192.168.3.2:24011, attached to remote volume '/mnt/sdc'.
[2011-11-29 18:37:57.203950] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-7: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.204077] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-1: Connected to 192.168.3.2:24010, attached to remote volume '/mnt/sdb'.
[2011-11-29 18:37:57.205230] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-9: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.205370] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-7: Connected to 192.168.3.2:24013, attached to remote volume '/mnt/sde'.
[2011-11-29 18:37:57.206095] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-5: Connected to 192.168.3.2:24012, attached to remote volume '/mnt/sdd'.
[2011-11-29 18:37:57.206860] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-9: Connected to 192.168.3.2:24014, attached to remote volume '/mnt/sdf'.
[2011-11-29 18:37:57.207677] I [client-handshake.c:1090:select_server_supported_programs] 0-test-client-11: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2011-11-29 18:37:57.209131] I [client-handshake.c:913:client_setvolume_cbk] 0-test-client-11: Connected to 192.168.3.2:24015, attached to remote volume '/mnt/sdg'.
[2011-11-29 18:37:57.213300] I [fuse-bridge.c:3339:fuse_graph_setup] 0-fuse: switched to graph 0
[2011-11-29 18:37:57.213538] I [fuse-bridge.c:2927:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16
[2011-11-29 18:37:57.214785] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-0: added root inode
[2011-11-29 18:37:57.223771] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-1: added root inode
[2011-11-29 18:37:57.223815] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-2: added root inode
[2011-11-29 18:37:57.223847] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-3: added root inode
[2011-11-29 18:37:57.224207] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-4: added root inode
[2011-11-29 18:37:57.224299] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-test-replicate-5: added root inode
[2011-11-29 18:37:59.706090] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:37:59.706222] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:37:59.707070] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.709197] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.710551] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.712251] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.712277] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:37:59.712295] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:37:59.713327] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.714566] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.715055] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:37:59.718001] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:37:59.718032] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:37:59.718986] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:15.326930] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:15.326998] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:15.356263] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:38:15.357185] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:38:15.360324] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.361557] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.363391] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.368037] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.368066] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:15.368081] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:15.373134] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.373666] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.376606] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:15.378826] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:15.378856] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:15.378871] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:56.418045] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:56.418207] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:38:56.424212] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:38:56.425218] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:38:56.428004] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.428483] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.430055] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.434384] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.434478] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:56.434581] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:56.438663] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.441256] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.444564] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.446613] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.446707] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:38:56.446814] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:38:56.449230] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.450378] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.451742] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.452876] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.631225] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.633071] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.638323] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.639472] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.651353] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.653039] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.664758] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.668635] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.681062] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.682423] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.743580] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.744993] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.756600] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.757805] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:38:56.765987] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:38:56.767405] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.983472] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:33.983620] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:33.988224] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:39:33.988741] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:39:33.990216] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:33.990312] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.991275] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:33.993757] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.994096] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:33.994160] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:39:33.998003] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:33.998334] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:33.999673] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:34.772] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:34.1130] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:34.1507] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:39:34.4248] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:34.5176] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:34.6302] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:34.7090] E [client3_1-fops.c:411:client3_1_stat_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.367727] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:43.368253] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:39:43.373007] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:39:43.375917] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:39:43.382030] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.382155] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.382461] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.384980] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.385099] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:43.385180] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:39:43.390265] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.393514] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.394545] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:39:43.397245] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:39:43.397339] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:39:43.397417] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:33.443714] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:33.444153] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:33.448884] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:40:33.449820] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:40:33.451464] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.453891] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.454991] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.457787] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.458849] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:40:33.458935] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:33.461319] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.461494] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.463903] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:33.466504] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:33.466599] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:40:33.467119] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:36.214909] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-0: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:36.215383] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-test-client-1: remote operation failed: Stale NFS file handle
[2011-11-29 18:40:36.223477] W [dht-common.c:177:dht_lookup_dir_cbk] 0-test-dht: /lost+found: gfid different on test-replicate-0
[2011-11-29 18:40:36.224517] I [dht-layout.c:581:dht_layout_normalize] 0-test-dht: found anomalies in /lost+found. holes=1 overlaps=0
[2011-11-29 18:40:36.228495] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.229136] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.229438] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.240941] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.241168] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child
[2011-11-29 18:40:36.242572] I [afr-transaction.c:945:afr_post_blocking_inodelk_cbk] 0-test-replicate-0: Blocking inodelks failed.
[2011-11-29 18:40:36.255641] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.255771] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.260899] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-0: remote operation failed: No such file or directory
[2011-11-29 18:40:36.265115] E [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-test-client-1: remote operation failed: No such file or directory
[2011-11-29 18:40:36.266560] I [afr-lk-common.c:960:afr_lock_blocking] 0-test-replicate-0: unable to lock on even one child

Comment 1 Amar Tumballi 2011-12-01 07:32:56 UTC
We acknowledge that this may be an issue now. But normally we recommend to use a directory inside your mount as export. (if /dev/sdb1 is mounted in /gfs_export/, don't use /gfs_export/ as brick dir, instead use /gfs_export/<VOLNAME> or something like that as export dir)

With that, in glusterfs exported directory, you won't have 'lost+found' directory itself.

Thanks for having the bug report, we will work on fix this glitch, but as there is a work around, we treat it as an enhancement.

Comment 2 ahazelwood 2011-12-01 11:39:10 UTC
Thanks for looking at this.  I wasn't sure how it should be marked, so enhancement makes sense. 

Please at least add this to the installation instructions as potentially a note as it would have saved me some time trying to track down.

Thanks.

Comment 3 Amar Tumballi 2012-04-17 18:43:55 UTC
Not working on fixing this, should surely need to document this behavior.

Comment 4 Anatoly Pugachev 2014-12-11 09:57:15 UTC
Can you please advice how to move from using block device mount as a brick to subdirectory in glusterfs. For example, move all data from /exports/brick1/ (/dev/sdb1 mounted) to /exports/brick1/datadir/ . thanks.

Comment 5 Kaleb KEITHLEY 2015-10-22 15:46:38 UTC
because of the large number of bugs filed against mainline version\ is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.


Note You need to log in before you can comment on or make changes to this bug.