Bug 1139998 - Renaming file while rebalance is in progress causes data loss
Summary: Renaming file while rebalance is in progress causes data loss
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: 3.4.5
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On: 969298 1130888 1140348 1146895
Blocks: 987422 glusterfs-3.4.6 1138395 1286059
TreeView+ depends on / blocked
 
Reported: 2014-09-10 07:07 UTC by Raghavendra G
Modified: 2015-11-27 10:26 UTC (History)
8 users (show)

Fixed In Version:
Clone Of: 1130888
Environment:
Last Closed: 2015-04-13 07:09:43 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)

Comment 1 Anand Avati 2014-09-10 08:38:20 UTC 
REVIEW: http://review.gluster.org/8679 (cluster/dht: introduce locking api.) posted (#1) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 2 Anand Avati 2014-09-10 08:38:23 UTC 
REVIEW: http://review.gluster.org/8680 (libglusterfs/syncop: implement inodelk) posted (#1) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 3 Anand Avati 2014-09-10 08:38:27 UTC 
REVIEW: http://review.gluster.org/8681 (cluster/dht: synchronize rename and file-migration) posted (#1) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 4 Anand Avati 2014-09-10 08:38:30 UTC 
REVIEW: http://review.gluster.org/8682 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#1) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 5 Anand Avati 2014-09-10 08:38:33 UTC 
REVIEW: http://review.gluster.org/8683 (cluster/dht: Rename should not fail post hardlink creation) posted (#1) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 6 Anand Avati 2014-09-10 08:38:39 UTC 
REVIEW: http://review.gluster.org/8685 (Cluster/DHT: Changing rename log severity) posted (#1) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 7 Anand Avati 2014-09-24 15:13:43 UTC 
REVIEW: http://review.gluster.org/8685 (Cluster/DHT: Changing rename log severity) posted (#2) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 8 Anand Avati 2014-09-24 15:13:58 UTC 
REVIEW: http://review.gluster.org/8683 (cluster/dht: Rename should not fail post hardlink creation) posted (#2) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 9 Anand Avati 2014-09-24 15:14:01 UTC 
REVIEW: http://review.gluster.org/8682 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#2) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 10 Anand Avati 2014-09-24 15:14:09 UTC 
REVIEW: http://review.gluster.org/8681 (cluster/dht: synchronize rename and file-migration) posted (#2) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 11 Anand Avati 2014-09-24 15:14:12 UTC 
REVIEW: http://review.gluster.org/8680 (libglusterfs/syncop: implement inodelk) posted (#2) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 12 Anand Avati 2014-09-24 15:14:16 UTC 
REVIEW: http://review.gluster.org/8679 (cluster/dht: introduce locking api.) posted (#2) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 13 Anand Avati 2014-09-25 12:00:32 UTC 
REVIEW: http://review.gluster.org/8685 (Cluster/DHT: Changing rename log severity) posted (#3) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 14 Anand Avati 2014-09-25 12:00:40 UTC 
REVIEW: http://review.gluster.org/8683 (cluster/dht: Rename should not fail post hardlink creation) posted (#3) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 15 Anand Avati 2014-09-25 12:00:44 UTC 
REVIEW: http://review.gluster.org/8682 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#3) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 16 Anand Avati 2014-09-25 12:00:47 UTC 
REVIEW: http://review.gluster.org/8681 (cluster/dht: synchronize rename and file-migration) posted (#3) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 17 Anand Avati 2014-09-25 12:00:51 UTC 
REVIEW: http://review.gluster.org/8680 (libglusterfs/syncop: implement inodelk) posted (#3) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 18 Anand Avati 2014-09-25 12:00:54 UTC 
REVIEW: http://review.gluster.org/8679 (cluster/dht: introduce locking api.) posted (#3) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 19 Anand Avati 2014-09-25 13:53:30 UTC 
REVIEW: http://review.gluster.org/8685 (Cluster/DHT: Changing rename log severity) posted (#4) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 20 Anand Avati 2014-09-25 13:53:33 UTC 
REVIEW: http://review.gluster.org/8683 (cluster/dht: Rename should not fail post hardlink creation) posted (#4) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 21 Anand Avati 2014-09-25 13:53:36 UTC 
REVIEW: http://review.gluster.org/8682 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#4) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 22 Anand Avati 2014-09-25 13:53:39 UTC 
REVIEW: http://review.gluster.org/8681 (cluster/dht: synchronize rename and file-migration) posted (#4) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 23 Anand Avati 2014-09-25 13:53:42 UTC 
REVIEW: http://review.gluster.org/8680 (libglusterfs/syncop: implement inodelk) posted (#4) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 24 Anand Avati 2014-09-25 13:53:45 UTC 
REVIEW: http://review.gluster.org/8679 (cluster/dht: introduce locking api.) posted (#4) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 25 Anand Avati 2014-10-09 05:39:59 UTC 
REVIEW: http://review.gluster.org/8685 (Cluster/DHT: Changing rename log severity) posted (#5) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 26 Anand Avati 2014-10-09 05:40:03 UTC 
REVIEW: http://review.gluster.org/8683 (cluster/dht: Rename should not fail post hardlink creation) posted (#5) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 27 Anand Avati 2014-10-09 05:40:07 UTC 
REVIEW: http://review.gluster.org/8682 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#5) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 28 Anand Avati 2014-10-09 05:40:10 UTC 
REVIEW: http://review.gluster.org/8681 (cluster/dht: synchronize rename and file-migration) posted (#5) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 29 Anand Avati 2014-10-09 05:40:13 UTC 
REVIEW: http://review.gluster.org/8680 (libglusterfs/syncop: implement inodelk) posted (#5) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 30 Anand Avati 2014-10-09 05:40:16 UTC 
REVIEW: http://review.gluster.org/8679 (cluster/dht: introduce locking api.) posted (#5) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 31 Anand Avati 2014-10-09 07:34:19 UTC 
REVIEW: http://review.gluster.org/8685 (Cluster/DHT: Changing rename log severity) posted (#6) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 32 Anand Avati 2014-10-09 07:34:22 UTC 
REVIEW: http://review.gluster.org/8683 (cluster/dht: Rename should not fail post hardlink creation) posted (#6) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 33 Anand Avati 2014-10-09 07:34:25 UTC 
REVIEW: http://review.gluster.org/8682 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#6) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 34 Anand Avati 2014-10-09 07:34:33 UTC 
REVIEW: http://review.gluster.org/8681 (cluster/dht: synchronize rename and file-migration) posted (#6) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 35 Anand Avati 2014-10-09 07:34:36 UTC 
REVIEW: http://review.gluster.org/8680 (libglusterfs/syncop: implement inodelk) posted (#6) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 36 Anand Avati 2014-10-09 07:34:40 UTC 
REVIEW: http://review.gluster.org/8679 (cluster/dht: introduce locking api.) posted (#6) for review on release-3.4 by Raghavendra G (rgowdapp)
Comment 37 Anand Avati 2014-10-20 14:55:14 UTC 
COMMIT: http://review.gluster.org/8679 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit fcd256faa951f459e344ced46f0cc409a5bbc147
Author: Raghavendra G <rgowdapp>
Date:   Mon Aug 11 10:14:18 2014 +0530

    cluster/dht: introduce locking api.
    
    Change-Id: I41389ba91951d3e63e617aa32cd0bee848261c72
    BUG: 1139998
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8521
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8679
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
Comment 38 Anand Avati 2014-10-20 14:55:35 UTC 
COMMIT: http://review.gluster.org/8680 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit 0ebe35ead483071ac3ff4b5c3802f40b1c719c8e
Author: Raghavendra G <rgowdapp>
Date:   Thu Aug 14 16:44:23 2014 +0530

    libglusterfs/syncop: implement inodelk
    
    Change-Id: Iea489157490b70cb2bb03576b0d4943c6d8f052d
    BUG: 1139998
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8522
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8680
    Reviewed-by: N Balachandran <nbalacha>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
Comment 39 Anand Avati 2014-10-20 14:55:44 UTC 
COMMIT: http://review.gluster.org/8681 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit c527449c0504828378f33ed32d1e7fe2e2c51c6f
Author: Raghavendra G <rgowdapp>
Date:   Wed Sep 10 09:05:22 2014 +0530

    cluster/dht: synchronize rename and file-migration
    
    Change-Id: I4f243c946f76d440680b651235f925e3d0ebf0fd
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8523
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    BUG: 1139998
    Reviewed-on: http://review.gluster.org/8681
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
Comment 40 Anand Avati 2014-10-20 14:55:52 UTC 
COMMIT: http://review.gluster.org/8682 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit 49c4106fc338ebfeef15d4800cb8c17e575d8d3e
Author: Shyam <srangana>
Date:   Wed Sep 10 09:23:15 2014 +0530

    cluster/dht: Treat linkto file rename failure as non-critial error
    
    It is a critical failure iff we fail to rename the cached file
    if the rename of the linkto failed, it is not a critical failure,
    and we do not want to lose the created hard link for the new
    name as that could have been read by other clients.
    
    NOTE: If another client is attempting the same oldname -> newname
    rename, and finds both file names as existing, and are hard links
    to each other, then FUSE would send in an unlink for oldname. In
    this time duration if we treat the linkto as a critical error and
    unlink the newname we created, we would have effectively lost the
    file to rename operations.
    
    Repercussions of treating this as a non-critical error is that
    we could leave behind a stale linkto file and/or not create the new
    linkto file, the second case would be rectified by a subsequent
    lookup, the first case by a rebalance, like for all stale linkto
    files
    
    Change-Id: Ia53ad8b43c3cf8f48ef5b43fd1fec4274e807556
    BUG: 1139998
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/8563
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8682
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
Comment 41 Anand Avati 2014-10-20 14:56:04 UTC 
COMMIT: http://review.gluster.org/8683 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit 3ade63522322d3c09b603a802eb1058cf4d0b50d
Author: Shyam <srangana>
Date:   Wed Sep 10 09:55:11 2014 +0530

    cluster/dht: Rename should not fail post hardlink creation
    
    In the rename path, we wind the creation of newname hardlink and
    linkto file in dst hashed a the same time. If the linkto creation
    fails, but the link creation succeeds, we enter the failure code
    and cleanup the created newname hardlink.
    
    In the interim if another client looks up newname and finds it as
    a hardlink from FUSE, it could send an unlink for oldname instead
    of a rename. This combined with the above cleanup code could end
    up losing all the files copies, and thereby losing data.
    
    This fix separates these steps into 2 parts, creating the linkto
    first and then the link file, so that post link file creation no
    failures would cleanup the newname file. If linkto fails then link
    is not attempted, thereby not polluting the name space with
    newname.
    
    Change-Id: I61da8e906060da16a31ea1076eec2f01fd617f44
    BUG: 1139998
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/8570
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8683
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
Comment 42 Anand Avati 2014-10-20 14:56:19 UTC 
COMMIT: http://review.gluster.org/8685 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit fe3e541ac559c975f7b27cb07834c572db1c4465
Author: Nithya Balachandran <nbalacha>
Date:   Wed Sep 3 10:42:47 2014 +0530

    Cluster/DHT: Changing rename log severity
    
    Changing log level for a rename message from debug
    to info to improve debuggability
    
    Change-Id: I53031fcf97fffd62095692477330ecde0cf47dcd
    BUG: 1139998
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8582
    Reviewed-by: Vijay Bellur <vbellur>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-on: http://review.gluster.org/8685
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
Note You need to log in before you can comment on or make changes to this bug.