Hide Forgot
Vikas's patch for this is undergoing testing. Looks promising as it fixes the ESTALEs for the test described below.
On file/dir creation, replicate returns inode from random subvolumes. This inode number would end up being different from the ino returned in stat or lookup resulting in a path resolution failure in unfs3booster. In response, unfs3booster returned an ESTALE to the NFS client. The simples way to reproduce it is: On the NFS client mount: $ mkdir mount/level1 $ mkdir mount/level1/level2 $ mkdir mount/level1/level2/level3 $ cd mount/level1/level2/level3 NOW RESTART UNFSD $ ls gives an Stale file handle error. Further instrumentation of unfs3 results in the following logs which show the problem. See inline comments: ============================================================ [root@client11 shehjart]# rm glusterfsd/booster.log -f ;LD_PRELOAD=/root/shehjart/glusterfsd/lib/glusterfs/libglusterfs-booster.so /root/shehjart/unfsd/sbin/unfsd -e /root/shehjart/exports -d UNFS3 unfsd 0.9.23 (C) 2009, Pascal Schmidt <unfs3-server> /testpath/: ip 192.168.101.0 mask 255.255.255.0 options 5 fh cache add: 27 1 /testpath/ entries 0 Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ FIRST MKDIR CALL @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ MKDIR Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 MKDIR: parent: /testpath/ MKDIR: newdir: /testpath//level1 mkdir succeeded fh_extend_type for /testpath//level1 Extending fh:FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Extending for: dev: 27, ino: 2836201479 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Note the above ino for the new directory @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Extended new fh:FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Done extending Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Looking for FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 1 Not found in cache Export point: /testpath/, levels: 1 FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 fh_rec entry: /testpath//. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//.. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//level1 stat buf: dev: 27, ino: 2836201479 Match found decomp raw: /testpath//level1 fh cache add: 27 2836201479 /testpath//level1 entries 1 Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1 Looking for FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2 Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1 MKDIR Looking for FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2 Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1 MKDIR: parent: /testpath//level1MKDIR: newdir: /testpath//level1/level2mkdir succeeded fh_extend_type for /testpath//level1/level2 Extending fh:FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Extending for: dev: 27, ino: 854163464 Extended new fh:FH Fields dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3 1 163 83 Done extending Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Extending fh:FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Extending for: dev: 27, ino: 2836201479 Extended new fh:FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2 Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1 Looking for FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2 Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1 Extending fh:FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Extending for: dev: 27, ino: 854163464 Extended new fh:FH Fields dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3 1 163 83 fh cache add: 27 854163464 /testpath//level1/level2 entries 2 Looking for FH Fields dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3 1 163 83 Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3 Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2 MKDIR Looking for FH Fields dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3 1 163 83 Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3 Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2 MKDIR: parent: /testpath//level1/level2MKDIR: newdir: /testpath//level1/level2/level3mkdir succeeded fh_extend_type for /testpath//level1/level2/level3 Extending fh:FH Fields dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3 1 163 83 Extending for: dev: 27, ino: 68242885 Extended new fh:FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 Done extending ^[ Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Looking for FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Extending fh:FH Fields dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1 1 Extending for: dev: 27, ino: 2836201479 Extended new fh:FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 fh cache add: 27 2836201479 /testpath//level1 entries 3 entry exists Looking for FH Fields dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2 1 163 Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 3 Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1 Looking for FH Fields dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3 1 163 83 Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3 Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2 Looking for FH Fields dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3 1 163 83 Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3 Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2 Looking for FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 3 Not found in cache Export point: /testpath/, levels: 1 FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 fh_rec entry: /testpath//. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//.. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//level1 stat buf: dev: 27, ino: 2836201479 ino hash 163, pos: 1, pos->ino: 163 Found component fh_rec entry: /testpath//level1/. stat buf: dev: 27, ino: 2836201479 ino hash 163, pos: 2, pos->ino: 83 fh_rec entry: /testpath//level1/.. stat buf: dev: 27, ino: 1 ino hash 1, pos: 2, pos->ino: 83 fh_rec entry: /testpath//level1/level2 stat buf: dev: 27, ino: 854163464 ino hash 83, pos: 2, pos->ino: 83 Found component fh_rec entry: /testpath//level1/level2/. stat buf: dev: 27, ino: 854163464 ino hash 83, pos: 3, pos->ino: 157 fh_rec entry: /testpath//level1/level2/.. stat buf: dev: 27, ino: 2836201479 ino hash 163, pos: 3, pos->ino: 157 fh_rec entry: /testpath//level1/level2/level3 stat buf: dev: 27, ino: 68242885 Match found decomp raw: /testpath//level1/level2/level3 fh cache add: 27 68242885 /testpath//level1/level2/level3 entries 3 Looked up in fh-cache dev: 27 ino: 68242885 path: /testpath//level1/level2/level3 Looking for FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 4 Looked up in fh-cache dev: 27 ino: 68242885 path: /testpath//level1/level2/level3 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ UNFSD RESTART @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [root@client11 shehjart]# rm glusterfsd/booster.log -f ;LD_PRELOAD=/root/shehjart/glusterfsd/lib/glusterfs/libglusterfs-booster.so /root/shehjart/unfsd/sbin/unfsd -e /root/shehjart/exports -d UNFS3 unfsd 0.9.23 (C) 2009, Pascal Schmidt <unfs3-server> /testpath/: ip 192.168.101.0 mask 255.255.255.0 options 5 Looking for FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 0 Not found in cache Export point: /testpath/, levels: 1 FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 fh_rec entry: /testpath//. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//.. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//level1 stat buf: dev: 27, ino: 5672402957 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ For directory /testpath/level1, note the changed inode number above. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ ino hash 68, pos: 1, pos->ino: 163 decomp raw: (null) @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Consequently, the file handle decomposition function above returns a failed path resolution @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Looked up in fh-cache dev: 27 ino: 68242885 path: (null) Looking for FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 0 Not found in cache Export point: /testpath/, levels: 1 FH Fields dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4 1 163 83 157 fh_rec entry: /testpath//. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//.. stat buf: dev: 27, ino: 1 ino hash 1, pos: 1, pos->ino: 163 fh_rec entry: /testpath//level1 stat buf: dev: 27, ino: 5672402957 ino hash 68, pos: 1, pos->ino: 163 decomp raw: (null) Looked up in fh-cache dev: 27 ino: 68242885 path: (null) Looked up in fh-cache dev: 27 ino: 68242885 path: (null)
PATCH: http://patches.gluster.com/patch/841 in master (replicate: Return ino from first subvolume on inode creation fops)
PATCH: http://patches.gluster.com/patch/840 in release-2.0 (replicate: Return ino from first subvolume on inode creation fops)
Fixed in release-2.0: http://git.gluster.com/?p=glusterfs.git;a=commit;h=65c03aa578340e78bd828f1f1aae991ef0cdc448 in mainline: http://git.gluster.com/?p=glusterfs.git;a=commit;h=fdbc5cf320fe85d128d498184135df10146047ba