Bug 761909 (GLUSTER-177)

Summary:	replicate: On file/dir creation, replicate returns inode from random subvolumes
Product:	[Community] GlusterFS	Reporter:	Shehjar Tikoo <shehjart>
Component:	replicate	Assignee:	Shehjar Tikoo <shehjart>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	low
Version:	mainline	CC:	gluster-bugs, vikas
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	---
Regression:	RTP	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Shehjar Tikoo 2009-07-30 07:43:27 UTC

Vikas's patch for this is undergoing testing. Looks promising as it fixes the ESTALEs for the test described below.

Comment 1 Shehjar Tikoo 2009-07-30 10:28:27 UTC

On file/dir creation, replicate returns inode from random subvolumes. This inode number would end up being different from the ino returned in stat or lookup resulting in a path resolution failure in unfs3booster.

In response, unfs3booster returned an ESTALE to the NFS client.

The simples way to reproduce it is:

On the NFS client mount:
$ mkdir mount/level1
$ mkdir mount/level1/level2
$ mkdir mount/level1/level2/level3
$ cd mount/level1/level2/level3
NOW RESTART UNFSD
$ ls
gives an Stale file handle error.


Further instrumentation of unfs3 results in the following logs which show the problem. See inline comments:

============================================================

[root@client11 shehjart]# rm glusterfsd/booster.log -f ;LD_PRELOAD=/root/shehjart/glusterfsd/lib/glusterfs/libglusterfs-booster.so /root/shehjart/unfsd/sbin/unfsd -e /root/shehjart/exports -d
UNFS3 unfsd 0.9.23 (C) 2009, Pascal Schmidt <unfs3-server>
/testpath/: ip 192.168.101.0 mask 255.255.255.0 options 5
fh cache add: 27 1 /testpath/ entries 0
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
FIRST MKDIR CALL
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
MKDIR
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
MKDIR: parent: /testpath/
MKDIR: newdir: /testpath//level1
mkdir succeeded
fh_extend_type for /testpath//level1
Extending fh:FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Extending for: dev: 27, ino: 2836201479

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Note the above ino for the new directory
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Extended new fh:FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Done extending
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Looking for FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 1
Not found in cache
Export point: /testpath/, levels: 1 
FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
fh_rec entry: /testpath//.
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163   
fh_rec entry: /testpath//..
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163   
fh_rec entry: /testpath//level1
stat buf: dev: 27, ino: 2836201479  
Match found
decomp raw: /testpath//level1
fh cache add: 27 2836201479 /testpath//level1 entries 1
Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1
Looking for FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2
Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1
MKDIR
Looking for FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2
Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1
MKDIR: parent: /testpath//level1MKDIR: newdir: /testpath//level1/level2mkdir succeeded
fh_extend_type for /testpath//level1/level2
Extending fh:FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Extending for: dev: 27, ino: 854163464
Extended new fh:FH Fields
dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3
1 163 83
Done extending
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Extending fh:FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Extending for: dev: 27, ino: 2836201479
Extended new fh:FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2
Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1
Looking for FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 2
Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1
Extending fh:FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Extending for: dev: 27, ino: 854163464
Extended new fh:FH Fields
dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3
1 163 83
fh cache add: 27 854163464 /testpath//level1/level2 entries 2
Looking for FH Fields
dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3
1 163 83
Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3
Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2
MKDIR
Looking for FH Fields
dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3
1 163 83
Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3
Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2
MKDIR: parent: /testpath//level1/level2MKDIR: newdir: /testpath//level1/level2/level3mkdir succeeded
fh_extend_type for /testpath//level1/level2/level3
Extending fh:FH Fields
dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3
1 163 83
Extending for: dev: 27, ino: 68242885
Extended new fh:FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
Done extending
^[


Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Looking for FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Extending fh:FH Fields
dev: 27 ino: 1 gen: 0 pwhash: 0 fh-len: 1
1
Extending for: dev: 27, ino: 2836201479
Extended new fh:FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
fh cache add: 27 2836201479 /testpath//level1 entries 3
entry exists
Looking for FH Fields
dev: 27 ino: 2836201479 gen: 0 pwhash: 0 fh-len: 2
1 163
Looking up in fhc: dev: 27, ino: 2836201479 fh-cache-entries: 3
Looked up in fh-cache dev: 27 ino: 2836201479 path: /testpath//level1
Looking for FH Fields
dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3
1 163 83
Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3
Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2
Looking for FH Fields
dev: 27 ino: 854163464 gen: 0 pwhash: 0 fh-len: 3
1 163 83
Looking up in fhc: dev: 27, ino: 854163464 fh-cache-entries: 3
Looked up in fh-cache dev: 27 ino: 854163464 path: /testpath//level1/level2
Looking for FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 3
Not found in cache
Export point: /testpath/, levels: 1 
FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
fh_rec entry: /testpath//.
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163   
fh_rec entry: /testpath//..
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163   
fh_rec entry: /testpath//level1
stat buf: dev: 27, ino: 2836201479  
ino hash 163, pos: 1, pos->ino: 163 
Found component
fh_rec entry: /testpath//level1/.   
stat buf: dev: 27, ino: 2836201479  
ino hash 163, pos: 2, pos->ino: 83  
fh_rec entry: /testpath//level1/..  
stat buf: dev: 27, ino: 1
ino hash 1, pos: 2, pos->ino: 83
fh_rec entry: /testpath//level1/level2
stat buf: dev: 27, ino: 854163464   
ino hash 83, pos: 2, pos->ino: 83   
Found component
fh_rec entry: /testpath//level1/level2/.
stat buf: dev: 27, ino: 854163464   
ino hash 83, pos: 3, pos->ino: 157  
fh_rec entry: /testpath//level1/level2/..
stat buf: dev: 27, ino: 2836201479  
ino hash 163, pos: 3, pos->ino: 157 
fh_rec entry: /testpath//level1/level2/level3
stat buf: dev: 27, ino: 68242885
Match found
decomp raw: /testpath//level1/level2/level3
fh cache add: 27 68242885 /testpath//level1/level2/level3 entries 3
Looked up in fh-cache dev: 27 ino: 68242885 path: /testpath//level1/level2/level3
Looking for FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 4
Looked up in fh-cache dev: 27 ino: 68242885 path: /testpath//level1/level2/level3


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                                  UNFSD RESTART
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@client11 shehjart]# rm glusterfsd/booster.log -f ;LD_PRELOAD=/root/shehjart/glusterfsd/lib/glusterfs/libglusterfs-booster.so /root/shehjart/unfsd/sbin/unfsd -e /root/shehjart/exports -d
UNFS3 unfsd 0.9.23 (C) 2009, Pascal Schmidt <unfs3-server>
/testpath/: ip 192.168.101.0 mask 255.255.255.0 options 5
Looking for FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 0
Not found in cache
Export point: /testpath/, levels: 1
FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
fh_rec entry: /testpath//.
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163
fh_rec entry: /testpath//..
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163
fh_rec entry: /testpath//level1
stat buf: dev: 27, ino: 5672402957

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
For directory /testpath/level1, note the changed inode number above.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


ino hash 68, pos: 1, pos->ino: 163
decomp raw: (null)



@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Consequently, the file handle decomposition function above returns a failed path resolution
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Looked up in fh-cache dev: 27 ino: 68242885 path: (null)
Looking for FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
Looking up in fhc: dev: 27, ino: 68242885 fh-cache-entries: 0
Not found in cache
Export point: /testpath/, levels: 1
FH Fields
dev: 27 ino: 68242885 gen: 0 pwhash: 0 fh-len: 4
1 163 83 157
fh_rec entry: /testpath//.
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163
fh_rec entry: /testpath//..
stat buf: dev: 27, ino: 1
ino hash 1, pos: 1, pos->ino: 163
fh_rec entry: /testpath//level1
stat buf: dev: 27, ino: 5672402957
ino hash 68, pos: 1, pos->ino: 163
decomp raw: (null)
Looked up in fh-cache dev: 27 ino: 68242885 path: (null)
Looked up in fh-cache dev: 27 ino: 68242885 path: (null)

Comment 2 Anand Avati 2009-07-30 16:00:58 UTC

PATCH: http://patches.gluster.com/patch/841 in master (replicate: Return ino from first subvolume on inode creation fops)

Comment 3 Anand Avati 2009-07-30 16:01:06 UTC

PATCH: http://patches.gluster.com/patch/840 in release-2.0 (replicate: Return ino from first subvolume on inode creation fops)

Comment 4 Shehjar Tikoo 2009-07-31 06:47:15 UTC

Fixed in release-2.0:
http://git.gluster.com/?p=glusterfs.git;a=commit;h=65c03aa578340e78bd828f1f1aae991ef0cdc448

in mainline:
http://git.gluster.com/?p=glusterfs.git;a=commit;h=fdbc5cf320fe85d128d498184135df10146047ba