Bug 1129527 - DHT :- data loss - file is missing on renaming same file from multiple client at same time
Summary: DHT :- data loss - file is missing on renaming same file from multiple client...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: 3.5.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Nithya Balachandran
QA Contact:
URL:
Whiteboard:
Depends On: 1117851 1138387 1139988
Blocks: glusterfs-3.5.3
TreeView+ depends on / blocked
 
Reported: 2014-08-13 05:25 UTC by Nithya Balachandran
Modified: 2014-11-21 16:14 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.5.3
Doc Type: Bug Fix
Doc Text:
Clone Of: 1117851
Environment:
Last Closed: 2014-11-21 16:01:48 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Comment 1 Anand Avati 2014-09-15 07:27:28 UTC
REVIEW: http://review.gluster.org/8714 (dht: fix rename race) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 2 Anand Avati 2014-09-15 07:30:42 UTC
REVIEW: http://review.gluster.org/8718 (dht: fix rename race) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 3 Anand Avati 2014-09-17 15:33:56 UTC
COMMIT: http://review.gluster.org/8714 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 97ada676f981705ebf2749327b5e2349e5028446
Author: Jeff Darcy <jdarcy>
Date:   Wed Sep 10 21:28:44 2014 +0530

    dht: fix rename race
    
    If two clients try to rename the same file at the same time, we
    sometimes end up with *no file at all* in either the old or new
    location.  That's kind of bad.  The culprit seems to be some overly
    aggressive cleanup code.  AFAICT, based on today's study of the code,
    the intent of the changed section is to remove any linkfile we might
    have created before the actual rename.  However, what we're removing
    might not be our extra link.  If we're racing with another client that's
    also doing a rename, it might be the only remaining link to the user's
    data.  The solution, which is good enough to pass this test but almost
    certainly still not complete, is to be more selective about when we do
    this unlink.  Now, we only do it if we know that, at some point, we did
    in fact create the link without error (notably ENOENT on the source or
    EEXIST on the destination) ourselves.
    
    Change-Id: I8d8cce150b6f8b372c9fb813c90be58d69f8eb7b
    BUG: 1129527
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: http://review.gluster.org/8269
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit 950f9d8abe714708ca62b86f304e7417127e1132)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8714

Comment 4 Anand Avati 2014-09-17 15:34:21 UTC
COMMIT: http://review.gluster.org/8716 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit c0cc62c144c560dcbfe9c89a139f51fd5274e33f
Author: Raghavendra G <rgowdapp>
Date:   Wed Sep 10 21:40:26 2014 +0530

    storage/posix: removing deleting entries in case of creation failures
    
    The code is not atomic enough to not to delete a dentry created by a
    prallel dentry creation operation.
    
    Change-Id: I9bd6d2aa9e7a1c0688c0a937b02a4b4f56d7aa3d
    BUG: 1129527
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8327
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit 45fbf99cb669e891a84a8228cef27973f5e774bf)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8716
    Reviewed-by: Niels de Vos <ndevos>

Comment 5 Anand Avati 2014-09-17 15:34:53 UTC
COMMIT: http://review.gluster.org/8718 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 21342c76f123df72d6e78fdca437e92e5d6a6841
Author: Nithya Balachandran <nbalacha>
Date:   Wed Sep 10 22:45:56 2014 +0530

    dht: fix rename race
    
    Additional check to check if we created the linkto
    file before deleting it in the rename cleanup function
    
    Change-Id: I919cd7cb24f948ba4917eb9cf50d5169bb730a67
    BUG: 1129527
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8338
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit df770496ba5ed6d2c72bcfc76ca9e816a08c383a)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8718
    Reviewed-by: Jeff Darcy <jdarcy>

Comment 6 Niels de Vos 2014-10-05 12:59:31 UTC
The first (and last?) Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-October/018990.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 7 Niels de Vos 2014-11-05 09:24:15 UTC
The second Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions have been made available on [2] to make testing easier.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019359.html
[2] http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.3beta2/

Comment 8 Niels de Vos 2014-11-21 16:01:48 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.3, please reopen this bug report.

glusterfs-3.5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/announce/2014-November/000042.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.