REVIEW: http://review.gluster.org/8015 (cluster/afr: Change ESTALE meaning in afr) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/8015 (cluster/afr: Change ESTALE meaning in afr.) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)
Extracting the test case from the (private) bug description for general visibility. 0) Create a 1x2 volume 1) Perform for i in {1..30}; do mkdir $i; tar xf glusterfs-3.5git.tar.gz -C $i& done 2) kill one of the bricks in the replica pair, while this is going on 3) After a while, kill all tar processes 4) Create a backup directory and move all 1..30 dirs inside 'backup' 5) Start the untar processes in 1) again 6) Bring up the brick. Now the estale failures are observed: tar: glusterfs-3.5.0/tests/bugs/886998/strict-readdir.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-824753-file-locker.c tar: glusterfs-3.5.0/tests/bugs/bug-824753-file-locker.c: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-949930.t tar: glusterfs-3.5.0/tests/bugs/bug-949930.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-862967.t tar: glusterfs-3.5.0/tests/bugs/bug-862967.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-884455.t tar: glusterfs-3.5.0/tests/bugs/bug-884455.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-000000.t tar: glusterfs-3.5.0/tests/bugs/bug-000000.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-765564.t tar: glusterfs-3.5.0/tests/bugs/bug-765564.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-892730.t tar: glusterfs-3.5.0/tests/bugs/bug-892730.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-904300.t tar: glusterfs-3.5.0/tests/bugs/bug-904300.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-961615.t tar: glusterfs-3.5.0/tests/bugs/bug-961615.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-808400.t tar: glusterfs-3.5.0/tests/bugs/bug-808400.t: Cannot open: Stale file handle glusterfs-3.5.0/tests/bugs/bug-860663.t
REVIEW: http://review.gluster.org/8052 (fuse: Don't set name when pargfid is unknown) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/8015 (cluster/afr: Change ESTALE meaning in afr.) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/8052 (fuse: Don't set name when pargfid is unknown.) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/8015 (cluster/afr: Change ESTALE meaning in afr) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/8052 (fuse: Don't set name when pargfid is unknown.) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/8015 (Fix resolution issues across fuse/server/afr) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/8015 committed in master by Vijay Bellur (vbellur) ------ commit 12b7797bfb143890ad4ca085332ec2f5e1ed08b8 Author: Pranith Kumar K <pkarampu> Date: Thu Jun 12 20:38:34 2014 +0530 Fix resolution issues across fuse/server/afr Problems with fuse/server: Fuse loc touch up sets loc->name even when pargfid is not known. Server lookup does (pargfid, name) based lookup when name is set ignoring the gfid. Because of this server resolver finds that the lookup came on (null-pargfid, name) and fails the lookup with EINVAL. Fix: Don't set loc->name in loc_touchup if the pargfid is not known. Did the same even for server-resolver Problem with afr: Lets say there is a directory hierarchy a/b/c/d on the mount and the user is cd'ed into the directory. Bring down one of the bricks of replica and remove all directories/files to simulate disk replacement on that brick. Now this brick is brought back up. Creates on the cd'ed directory fail with ESTALE. Basically before sending a create of 'f' inside 'd', fuse sends a lookup to make sure the file is not present. On one of the bricks 'd' is present and 'f' is not so it sends ENOENT as response. On the new brick 'd' itself is not present. So it sends ESTALE. In afr ESTALE is considered to be special errno on witnessing which lookup has to fail. And ESTALE is given more priority than ENOENT. Due to these reasons lookup fails with ESTALE rather than ENOENT. Since lookup didn't fail with ENOENT, 'create' can't be issued so the command is failed with ESTALE. Solution: Afr needs to consider ESTALE errno normally and ENOENT needs to be given more priority so that operations like create can proceed even when only one of the brick is up and running. Whenever client xlator identifies that gfid-changed, it sets that information in lookup xdata. Afr uses this information to fail the lookup with ESTALE so that top xlator can send fresh lookup. Change-Id: Ica6ce01baef08620154050a635e6f97d51029ef6 BUG: 1106408 Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/8015 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html [2] http://supercolony.gluster.org/pipermail/gluster-users/
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report. glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html [2] http://supercolony.gluster.org/mailman/listinfo/gluster-users