+++ This bug was initially created as a clone of Bug #1500269 +++

Description of problem:

Consider two programs:

rename:
=======
while true; do touch newfile; mv newfile file; done

open:
=====
keeps opening an fd on "file"

If I run these two programs in parallel (after making sure rename results in "file"), open on existing file "file" fails with ENOENT. Since "file" always existed and rename is atomic open should not fail with ENOENT.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Raghavendra G on 2017-10-10 06:00 EDT ---



--- Additional comment from Raghavendra G on 2017-10-10 06:00 EDT ---



--- Additional comment from Raghavendra G on 2017-10-10 06:04:15 EDT ---

Consider rename (index.new, store.idx) and open (store.idx) being executed in parallel. When we break down operations following sequence is possible.

* lookup (store.idx) - as part of open(store.idx) returns gfid1 as the result.
* rename (index.new, store.idx) changes gfid of store.idx to gfid2. Note that gfid2 was the nodeid of index.new. Since rename is successful, gfid2 is associated with store.idx.
* open (store.idx) resumes and issues open fop to glusterfs with gfid1. open in glusterfs fails as gfid1 doesn't exist and the error returned by glusterfs to kernel-fuse is ENOENT.
* kernel passes back the same error to application as a result to open.

This error could've been prevented if kernel retries open with gfid2. Interestingly kernel do retry open when it receives ESTALE error.

From kernel source code fs/namei.c:

<do_filep_open>

        filp = path_openat(&nd, op, flags | LOOKUP_RCU);
        if (unlikely(filp == ERR_PTR(-ECHILD)))
                filp = path_openat(&nd, op, flags);

        if (unlikely(filp == ERR_PTR(-ESTALE)))           <==== NOTE HERE
                filp = path_openat(&nd, op, flags | LOOKUP_REVAL);

</do_filep_open>

However that this hypothesis is _VALID ONLY IF_ kernel doesn't maintain the atomicity  of (lookup + open) in do_sys_open. If it is not maintaining atomicity, but instead relying on ESTALE error, this could be solved by passing back ESTALE as the error when we find that gfid is missing.

--- Additional comment from Raghavendra G on 2017-10-10 06:09:36 EDT ---

To verify the hypothesis I made following fixes

* reverted [1]
* converted ENOENT error to ESTALE in fuse_fd_cbk as open is an operation on gfid and correct error for a missing gfid is ESTALE. Note that some cases brick can return ENOENT for missing gfid. This can happen if resolver in protocol/server successfully resolves gfid, but the fop (open in this example) fails in storage/posix due to gfid-handle missing in .glusterfs

On applying these two fixes and rerunning the above test - open and rename.sh in infinite loop in parallel on the same mount - I was not able to hit the issue.

However if I run open and rename (c program), the issue is reproducible even with fixes. The explanation I could come up with is that,
* Kernel retries only once on getting an ESTALE error during open (as can be seen in do_filep_open code in kernel VFS)
* Binary "rename" generated by C program is fast enough that it could do more than one rename in the window of single "open".

This explanation is bit counter intuitive given that the program rename has two syscalls and the program open has one syscall. But, that's the best explanation I can come up with for now.

[1] http://review.gluster.org/13816

--- Additional comment from Worker Ant on 2017-10-10 06:26:08 EDT ---

REVIEW: https://review.gluster.org/18463 (Revert "mount/fuse: report ESTALE as ENOENT") posted (#1) for review on master by Raghavendra G (rgowdapp)

--- Additional comment from Worker Ant on 2017-10-13 10:33:20 EDT ---

REVIEW: https://review.gluster.org/18463 (Revert "mount/fuse: report ESTALE as ENOENT") posted (#2) for review on master by Raghavendra G (rgowdapp)

--- Additional comment from Worker Ant on 2017-10-13 10:33:23 EDT ---

REVIEW: https://review.gluster.org/18521 (mount/fuse: never fail open(dir) with ENOENT) posted (#1) for review on master by Raghavendra G (rgowdapp)

--- Additional comment from Worker Ant on 2017-10-17 01:14:45 EDT ---

COMMIT: https://review.gluster.org/18463 committed in master by Raghavendra G (rgowdapp) 
------
commit 019a55e708375d2b1e576fcc948a691bcdc5c749
Author: Raghavendra G <rgowdapp>
Date:   Tue Oct 10 11:29:04 2017 +0530

    Revert "mount/fuse: report ESTALE as ENOENT"
    
    This reverts commit 26d16b90ec7f8acbe07e56e8fe1baf9c9fa1519e.
    
    Consider rename (index.new, store.idx) and open (store.idx) being
    executed in parallel. When we break down operations following sequence
    is possible.
    
    * lookup (store.idx) - as part of open(store.idx) returns gfid1 as the
      result.
    * rename (index.new, store.idx) changes gfid of store.idx to
      gfid2. Note that gfid2 was the nodeid of index.new. Since rename is
      successful, gfid2 is associated with store.idx.
    * open (store.idx) resumes and issues open fop to glusterfs with
      gfid1. open in glusterfs fails as gfid1 doesn't exist and the error
      returned by glusterfs to kernel-fuse is ENOENT.
    * kernel passes back the same error to application as a result to
      open.
    
    This error could've been prevented if kernel retries open with
    gfid2. Interestingly kernel do retry open when it receives ESTALE
    error. Even though failure to find gfid resulted in ESTALE error,
    commit 26d16b90ec7f8acb converted that error to ENOENT while sending
    an error reply to kernel. This prevented kernel from retrying open
    resulting in error.
    
    Change-Id: I2e752ca60dd8af1b989dd1d29c7b002ee58440b4
    BUG: 1500269
    Signed-off-by: Raghavendra G <rgowdapp>

--- Additional comment from Worker Ant on 2017-10-17 01:15:36 EDT ---

REVIEW: https://review.gluster.org/18521 (mount/fuse: never fail open(dir) with ENOENT) posted (#2) for review on master by Raghavendra G (rgowdapp)

--- Additional comment from Worker Ant on 2017-10-17 01:31:47 EDT ---

COMMIT: https://review.gluster.org/18521 committed in master by Raghavendra G (rgowdapp) 
------
commit fb4b914ce84bc83a5f418719c5ba7c25689a9251
Author: Raghavendra G <rgowdapp>
Date:   Fri Oct 13 20:00:47 2017 +0530

    mount/fuse: never fail open(dir) with ENOENT
    
    open(dir) being an operation on inode should never fail with
    ENOENT. If gfid is not present, the appropriate error is ESTALE. This
    will enable kernel to retry open after a revalidate lookup.
    
    Change-Id: I8d07d2ebb5a0da6c3ea478317442cb42f1797a4b
    BUG: 1500269
    Signed-off-by: Raghavendra G <rgowdapp>

--- Additional comment from Shyamsundar on 2017-12-08 12:43:00 EST ---

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.13.0, please open a new bug report.

glusterfs-3.13.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-December/000087.html
[2] https://www.gluster.org/pipermail/gluster-users/