Bug 1140348 - Renaming file while rebalance is in progress causes data loss
Summary: Renaming file while rebalance is in progress causes data loss
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: 3.5.2
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
Assignee: Nithya Balachandran
QA Contact:
URL:
Whiteboard:
Depends On: 969298 1130888
Blocks: 987422 glusterfs-3.5.3 1138395 1139998 1146895 1286059
TreeView+ depends on / blocked
 
Reported: 2014-09-10 17:51 UTC by Nithya Balachandran
Modified: 2015-11-27 10:26 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.5.3
Doc Type: Bug Fix
Doc Text:
Clone Of: 1130888
Environment:
Last Closed: 2014-11-21 16:02:33 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Comment 1 Anand Avati 2014-09-15 07:31:10 UTC
REVIEW: http://review.gluster.org/8722 (cluster/dht: introduce locking api.) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 2 Anand Avati 2014-09-15 07:31:14 UTC
REVIEW: http://review.gluster.org/8723 (libglusterfs/syncop: implement inodelk) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 3 Anand Avati 2014-09-15 07:31:17 UTC
REVIEW: http://review.gluster.org/8724 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 4 Anand Avati 2014-09-15 07:31:25 UTC
REVIEW: http://review.gluster.org/8726 (cluster/dht: synchronize rename and file-migration) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 5 Anand Avati 2014-09-15 07:31:28 UTC
REVIEW: http://review.gluster.org/8727 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 6 Anand Avati 2014-09-15 07:31:32 UTC
REVIEW: http://review.gluster.org/8728 (cluster/dht: Rename should not fail post hardlink creation) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 7 Anand Avati 2014-09-15 07:31:44 UTC
REVIEW: http://review.gluster.org/8730 (Cluster/DHT: Changing rename log severity) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)

Comment 8 Anand Avati 2014-09-23 08:53:28 UTC
REVIEW: http://review.gluster.org/8727 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)

Comment 9 Anand Avati 2014-09-23 08:53:32 UTC
REVIEW: http://review.gluster.org/8726 (cluster/dht: synchronize rename and file-migration) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)

Comment 10 Anand Avati 2014-09-23 08:53:48 UTC
REVIEW: http://review.gluster.org/8724 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)

Comment 11 Anand Avati 2014-09-23 08:53:51 UTC
REVIEW: http://review.gluster.org/8723 (libglusterfs/syncop: implement inodelk) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)

Comment 12 Anand Avati 2014-09-23 08:53:55 UTC
REVIEW: http://review.gluster.org/8722 (cluster/dht: introduce locking api.) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)

Comment 13 Anand Avati 2014-09-23 08:54:08 UTC
REVIEW: http://review.gluster.org/8730 (Cluster/DHT: Changing rename log severity) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)

Comment 14 Anand Avati 2014-09-23 08:54:14 UTC
REVIEW: http://review.gluster.org/8728 (cluster/dht: Rename should not fail post hardlink creation) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)

Comment 15 Anand Avati 2014-09-24 09:01:02 UTC
REVIEW: http://review.gluster.org/8727 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)

Comment 16 Anand Avati 2014-09-24 09:01:05 UTC
REVIEW: http://review.gluster.org/8726 (cluster/dht: synchronize rename and file-migration) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)

Comment 17 Anand Avati 2014-09-24 09:01:11 UTC
REVIEW: http://review.gluster.org/8724 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)

Comment 18 Anand Avati 2014-09-24 09:01:14 UTC
REVIEW: http://review.gluster.org/8723 (libglusterfs/syncop: implement inodelk) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)

Comment 19 Anand Avati 2014-09-24 09:01:17 UTC
REVIEW: http://review.gluster.org/8722 (cluster/dht: introduce dht locking api.) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)

Comment 20 Anand Avati 2014-09-24 09:01:30 UTC
REVIEW: http://review.gluster.org/8730 (Cluster/DHT: Changing rename log severity) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)

Comment 21 Anand Avati 2014-09-24 09:01:33 UTC
REVIEW: http://review.gluster.org/8728 (cluster/dht: Rename should not fail post hardlink creation) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)

Comment 22 Anand Avati 2014-10-01 10:33:55 UTC
COMMIT: http://review.gluster.org/8722 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 53010998dc6c7bb5268cd1c124887ce9d9a3f032
Author: Raghavendra G <rgowdapp>
Date:   Wed Sep 10 23:22:03 2014 +0530

    cluster/dht: introduce dht locking api.
    
    Change-Id: I41389ba91951d3e63e617aa32cd0bee848261c72
    BUG: 1140348
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8521
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit a1fe3d72e373bf0deaed152842d12d94bb9129dc)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8722
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Reviewed-by: Niels de Vos <ndevos>

Comment 23 Anand Avati 2014-10-01 10:34:18 UTC
COMMIT: http://review.gluster.org/8723 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 16cc2a73d775d4f013f7abd68db547fd512f82f4
Author: Raghavendra G <rgowdapp>
Date:   Wed Sep 10 23:25:27 2014 +0530

    libglusterfs/syncop: implement inodelk
    
    Change-Id: Iea489157490b70cb2bb03576b0d4943c6d8f052d
    BUG: 1140348
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8522
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit 172100209e531e94168d0974bfb0995704d9dddd)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8723
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Reviewed-by: Niels de Vos <ndevos>

Comment 24 Anand Avati 2014-10-01 10:34:35 UTC
COMMIT: http://review.gluster.org/8724 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 4bff3e0f63b8e5c9dd286f31d52cb5de05eb71b7
Author: Raghavendra G <rgowdapp>
Date:   Wed Sep 10 23:30:03 2014 +0530

    cluster/dht: invoke callback when there are no locks to be unlocked.
    
    Change-Id: I375cb68f1075c2d58cf9d09ed6bd5e2746e1637d
    BUG: 1140348
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8549
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: N Balachandran <nbalacha>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit a1b02e53a5fdf706290ce143fbbf8a09845105d0)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8724
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Reviewed-by: Niels de Vos <ndevos>

Comment 25 Anand Avati 2014-10-01 10:35:58 UTC
COMMIT: http://review.gluster.org/8726 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit ab29e0fed53f4628281be6e8290546fb88f4f6d2
Author: Raghavendra G <rgowdapp>
Date:   Wed Sep 10 23:44:48 2014 +0530

    cluster/dht: synchronize rename and file-migration
    
    Change-Id: I4f243c946f76d440680b651235f925e3d0ebf0fd
    BUG: 1140348
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/8523
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit 21c8946b0bc05d0bc8f84906e16b8c2cbca4c9f9)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8726
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Reviewed-by: Niels de Vos <ndevos>

Comment 26 Anand Avati 2014-10-01 10:36:15 UTC
COMMIT: http://review.gluster.org/8727 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 58f1273c0831a0c0ac99f4bc66461ec88a112f62
Author: Shyam <srangana>
Date:   Wed Sep 10 23:58:35 2014 +0530

    cluster/dht: Treat linkto file rename failure as non-critial error
    
    It is a critical failure iff we fail to rename the cached file
    if the rename of the linkto failed, it is not a critical failure,
    and we do not want to lose the created hard link for the new
    name as that could have been read by other clients.
    
    NOTE: If another client is attempting the same oldname -> newname
    rename, and finds both file names as existing, and are hard links
    to each other, then FUSE would send in an unlink for oldname. In
    this time duration if we treat the linkto as a critical error and
    unlink the newname we created, we would have effectively lost the
    file to rename operations.
    
    Repercussions of treating this as a non-critical error is that
    we could leave behind a stale linkto file and/or not create the new
    linkto file, the second case would be rectified by a subsequent
    lookup, the first case by a rebalance, like for all stale linkto
    files
    
    Change-Id: Ia53ad8b43c3cf8f48ef5b43fd1fec4274e807556
    BUG: 1140348
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/8563
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit 890ab583a519b3b189a61c5fd563b4326836b988)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8727
    Reviewed-by: Niels de Vos <ndevos>

Comment 27 Anand Avati 2014-10-01 10:36:39 UTC
COMMIT: http://review.gluster.org/8728 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 5f31af8527e7f1f62e2c80fdf25f6d2cf71e49f1
Author: Shyam <srangana>
Date:   Thu Sep 11 10:52:57 2014 +0530

    cluster/dht: Rename should not fail post hardlink creation
    
    In the rename path, we wind the creation of newname hardlink and
    linkto file in dst hashed a the same time. If the linkto creation
    fails, but the link creation succeeds, we enter the failure code
    and cleanup the created newname hardlink.
    
    In the interim if another client looks up newname and finds it as
    a hardlink from FUSE, it could send an unlink for oldname instead
    of a rename. This combined with the above cleanup code could end
    up losing all the files copies, and thereby losing data.
    
    This fix separates these steps into 2 parts, creating the linkto
    first and then the link file, so that post link file creation no
    failures would cleanup the newname file. If linkto fails then link
    is not attempted, thereby not polluting the name space with
    newname.
    
    Change-Id: I61da8e906060da16a31ea1076eec2f01fd617f44
    BUG: 1140348
    Signed-off-by: Shyam <srangana>
    Reviewed-on: http://review.gluster.org/8570
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-by: Vijay Bellur <vbellur>
    (cherry picked from commit 4ce3db8e508e715a43352b082e861fd0e729951f)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8728
    Reviewed-by: Niels de Vos <ndevos>

Comment 28 Anand Avati 2014-10-01 10:36:55 UTC
COMMIT: http://review.gluster.org/8730 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 53cbf3146dacdd68f0a24ce6b2f924c77d79a411
Author: Nithya Balachandran <nbalacha>
Date:   Thu Sep 11 10:56:49 2014 +0530

    Cluster/DHT: Changing rename log severity
    
    Changing the log level for a rename message from debug
    to info to improve debuggability
    
    Change-Id: I53031fcf97fffd62095692477330ecde0cf47dcd
    BUG: 1140348
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8582
    Reviewed-by: Vijay Bellur <vbellur>
    Tested-by: Gluster Build System <jenkins.com>
    (cherry picked from commit c087e5f634a0b2262118d61ab9c1d5c8e18c8819)
    Signed-off-by: Nithya Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/8730
    Reviewed-by: venkatesh somyajulu <vsomyaju>
    Reviewed-by: Niels de Vos <ndevos>

Comment 29 Niels de Vos 2014-10-05 13:00:03 UTC
The first (and last?) Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-October/018990.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 30 Niels de Vos 2014-11-05 09:24:42 UTC
The second Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions have been made available on [2] to make testing easier.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019359.html
[2] http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.3beta2/

Comment 31 Niels de Vos 2014-11-21 16:02:33 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.3, please reopen this bug report.

glusterfs-3.5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/announce/2014-November/000042.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.