+++ This bug was initially created as a clone of Bug #1130888 +++ Description of problem: while rebalance is in progress renaming the files causes data loss Steps to Reproduce: 1.created a 2 brick distribute volume 2. mounted the volume and created 10000 files 3. add-brick and initiate rebalance, while rebalance is in progress rename all the files for i in {1..1000} do mv $i new$i done Actual results: on the mount point we can see the messages like [root@localhost mymount]# for i in {1..10000}; do mv $i new$i ; done mv: cannot move `1747' to `new1747': File exists mv: cannot move `2485' to `new2485': File exists mv: cannot move `2577' to `new2577': File exists mv: cannot move `2586' to `new2586': File exists mv: cannot move `3455' to `new3455': Structure needs cleaning mv: cannot move `3626' to `new3626': No such file or directory mv: cannot move `3829' to `new3829': No such file or directory mv: cannot move `3830' to `new3830': No such file or directory mv: cannot move `3944' to `new3944': Structure needs cleaning mv: cannot move `3963' to `new3963': Structure needs cleaning mv: cannot move `4180' to `new4180': Structure needs cleaning mv: cannot move `4506' to `new4506': Structure needs cleaning mv: cannot move `4591' to `new4591': Structure needs cleaning mv: cannot move `4601' to `new4601': Structure needs cleaning mv: cannot move `4602' to `new4602': Structure needs cleaning mv: cannot move `4611' to `new4611': Structure needs cleaning mv: cannot move `4644' to `new4644': No such file or directory mv: cannot move `4709' to `new4709': No such file or directory mv: cannot move `4814' to `new4814': No such file or directory mv: cannot move `4835' to `new4835': Structure needs cleaning mv: cannot move `4852' to `new4852': No such file or directory mv: cannot move `5009' to `new5009': No such file or directory the number files has been decreased [root@localhost mymount]# ls | wc -l 9922 mnt log says ================ 02:06.250800] W [dht-rename.c:482:dht_rename_cbk] 1-dist1-dht: /4852: rename on dist1-client-0 failed (No such file or directory) [2013-05-31 12:02:06.251172] W [fuse-bridge.c:1528:fuse_rename_cbk] 0-glusterfs-fuse: 103658: /4852 -> /new4852 => -1 (No such file o r directory) [2013-05-31 12:02:06.314825] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4858 on dist1-client-2 (following linkfile) reached link [2013-05-31 12:02:06.316650] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-2: remote operation failed: File exists. Pat h: /4858 (00000000-0000-0000-0000-000000000000) [2013-05-31 12:02:06.930730] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4909 on dist1-client-0 (following linkfile) reached link [2013-05-31 12:02:06.931329] W [dht-common.c:983:dht_lookup_everywhere_cbk] 1-dist1-dht: multiple subvolumes (dist1-client-1 and dist 1-client-0) have file /4909 (preferably rename the file in the backend, and do a fresh lookup) [2013-05-31 12:02:06.933437] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-0: remote operation failed: File exists. Pat h: /4909 (00000000-0000-0000-0000-000000000000) [2013-05-31 12:02:06.977724] W [dht-rename.c:334:dht_rename_unlink_cbk] 1-dist1-dht: /4912: unlink on dist1-client-1 failed (No such file or directory) [2013-05-31 12:02:07.022107] I [dht-common.c:997:dht_lookup_everywhere_cbk] 1-dist1-dht: deleting stale linkfile /4916 on dist1-clien t-2 [2013-05-31 12:02:07.282241] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4939 on dist1-client-0 (following linkfile) reached link [2013-05-31 12:02:07.283273] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-0: remote operation failed: File exists. Pat h: /4939 (00000000-0000-0000-0000-000000000000) [2013-05-31 12:02:07.284610] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4939 on dist1-client-0 (following linkfile) reached link [2013-05-31 12:02:07.288314] W [client3_1-fops.c:258:client3_1_mknod_cbk] 1-dist1-client-0: remote operation failed: File exists. Path: /4939 (00000000-0000-0000-0000-000000000000) [2013-05-31 12:02:07.325669] I [dht-common.c:1103:dht_lookup_linkfile_cbk] 1-dist1-dht: lookup of /4942 on dist1-client-0 (following linkfile) reached link --- Additional comment from shylesh on 2013-05-31 03:06:30 EDT --- [root@anshi1 ~]# gluster v info dist1 Volume Name: dist1 Type: Distribute Volume ID: ed55b825-0805-49c8-873c-8447681e687c Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 10.70.35.213:/brick2/dist1 Brick2: 10.70.35.230:/brick2/dist2 Brick3: 10.70.35.213:/brick2/dist3 --- Additional comment from on 2014-08-11 07:59:23 EDT --- In rebalance logs, we saw many files migrated from cached to hashed. After migration file should be from data-name by rebalance. But unlinks fails. It means that rename came in between and caused data loss. Race could be something like this: Src-cacehd Dst-hashed A A(linkto) mv A B came and and A should get renamed to B at src_cached and A(linkto) should get deleted. Migration migrated the A(Linkto) A(data) file from A to B. Rename case rename A->B unlink A So we are left with renamed linkto file.
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#2) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8609 (cluster/dht: introduce locking api.) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
COMMIT: http://review.gluster.org/8609 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit a325fef8c8f4ca35d65e3891cf72519e925e69f4 Author: Raghavendra G <rgowdapp> Date: Thu Sep 4 14:11:14 2014 -0400 cluster/dht: introduce locking api. Change-Id: I41389ba91951d3e63e617aa32cd0bee848261c72 BUG: 1138395 Signed-off-by: Raghavendra G <rgowdapp> Reviewed-on-master: http://review.gluster.org/8521 Reviewed-by: Pranith Kumar Karampuri <pkarampu> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/8609 Reviewed-by: Jeff Darcy <jdarcy>
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#3) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8611 (cluster/dht: invoke callback when there are no locks to be unlocked.) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8610 (libglusterfs/syncop: implement inodelk) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8614 (cluster/dht: Treat linkto file rename failure as non-critial error) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8613 (cluster/dht: synchronize rename and file-migration) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
COMMIT: http://review.gluster.org/8610 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit b0dfe8da3fdda31b57177a08de8acec31ac749bf Author: Raghavendra G <rgowdapp> Date: Thu Sep 4 14:12:39 2014 -0400 libglusterfs/syncop: implement inodelk Change-Id: Iea489157490b70cb2bb03576b0d4943c6d8f052d BUG: 1138395 Signed-off-by: Raghavendra G <rgowdapp> Reviewed-on-master: http://review.gluster.org/8522 Reviewed-by: Pranith Kumar Karampuri <pkarampu> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/8610
COMMIT: http://review.gluster.org/8611 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit 9ace73a2dd1db0359430a107e9ebc2a0dcb835d2 Author: Raghavendra G <rgowdapp> Date: Thu Sep 4 14:14:01 2014 -0400 cluster/dht: invoke callback when there are no locks to be unlocked. Change-Id: I375cb68f1075c2d58cf9d09ed6bd5e2746e1637d BUG: 1138395 Signed-off-by: Raghavendra G <rgowdapp> Reviewed-on-master: http://review.gluster.org/8549 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: N Balachandran <nbalacha> Reviewed-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/8611
COMMIT: http://review.gluster.org/8613 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit 0ae4768d35e76dee30b455e75a4b2552e4abb232 Author: Raghavendra G <rgowdapp> Date: Thu Sep 4 14:16:17 2014 -0400 cluster/dht: synchronize rename and file-migration Change-Id: I4f243c946f76d440680b651235f925e3d0ebf0fd BUG: 1138395 Signed-off-by: Raghavendra G <rgowdapp> Reviewed-on-master: http://review.gluster.org/8523 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/8613
COMMIT: http://review.gluster.org/8614 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit fa599ee2de35d4538ae395935be2fbe8184f0806 Author: Shyam <srangana> Date: Thu Sep 4 14:17:48 2014 -0400 cluster/dht: Treat linkto file rename failure as non-critial error It is a critical failure iff we fail to rename the cached file if the rename of the linkto failed, it is not a critical failure, and we do not want to lose the created hard link for the new name as that could have been read by other clients. NOTE: If another client is attempting the same oldname -> newname rename, and finds both file names as existing, and are hard links to each other, then FUSE would send in an unlink for oldname. In this time duration if we treat the linkto as a critical error and unlink the newname we created, we would have effectively lost the file to rename operations. Repercussions of treating this as a non-critical error is that we could leave behind a stale linkto file and/or not create the new linkto file, the second case would be rectified by a subsequent lookup, the first case by a rebalance, like for all stale linkto files Change-Id: Ia53ad8b43c3cf8f48ef5b43fd1fec4274e807556 BUG: 1138395 Signed-off-by: Shyam <srangana> Reviewed-on-master: http://review.gluster.org/8563 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy> Reviewed-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/8614
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#4) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
REVIEW: http://review.gluster.org/8615 (cluster/dht: Rename should not fail post hardlink creation) posted (#6) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
COMMIT: http://review.gluster.org/8615 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit a6123ebfdb67e71eab859f94fb14660e9e3701c6 Author: Shyam <srangana> Date: Thu Sep 4 14:21:10 2014 -0400 cluster/dht: Rename should not fail post hardlink creation In the rename path, we wind the creation of newname hardlink and linkto file in dst hashed a the same time. If the linkto creation fails, but the link creation succeeds, we enter the failure code and cleanup the created newname hardlink. In the interim if another client looks up newname and finds it as a hardlink from FUSE, it could send an unlink for oldname instead of a rename. This combined with the above cleanup code could end up losing all the files copies, and thereby losing data. This fix separates these steps into 2 parts, creating the linkto first and then the link file, so that post link file creation no failures would cleanup the newname file. If linkto fails then link is not attempted, thereby not polluting the name space with newname. Change-Id: I61da8e906060da16a31ea1076eec2f01fd617f44 BUG: 1138395 Signed-off-by: Shyam <srangana> Reviewed-on-master: http://review.gluster.org/8570 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy> Reviewed-by: Vijay Bellur <vbellur> Reviewed-on: http://review.gluster.org/8615
REVIEW: http://review.gluster.org/8626 (Cluster/DHT: Changing rename log severity) posted (#5) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
COMMIT: http://review.gluster.org/8626 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit 11485955083762d1c266dfae728a1abfd27aea63 Author: Nithya Balachandran <nbalacha> Date: Fri Sep 5 10:50:51 2014 -0400 Cluster/DHT: Changing rename log severity Changing log level for a rename message from debug to info to improve debuggability Change-Id: I53031fcf97fffd62095692477330ecde0cf47dcd BUG: 1138395 Signed-off-by: Nithya Balachandran <nbalacha> Reviewed-on-master: http://review.gluster.org/8582 Reviewed-by: Vijay Bellur <vbellur> Tested-by: Gluster Build System <jenkins.com> Signed-off-by: Shyam <srangana> Reviewed-on: http://review.gluster.org/8626
REVIEW: http://review.gluster.org/8782 (syncop: Invoke dict_unref() in inodelk only if dictionary is not NULL) posted (#1) for review on release-3.6 by Shyamsundar Ranganathan (srangana)
COMMIT: http://review.gluster.org/8782 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit 10d45b9783b751c1f78b670281fd1d57ae377642 Author: Vijay Bellur <vbellur> Date: Fri Sep 19 10:04:22 2014 -0400 syncop: Invoke dict_unref() in inodelk only if dictionary is not NULL In the absence of this check, logs can get flooded with messages like this when rebalance is run: [2014-09-04 17:48:07.148262] W [dict.c:480:dict_unref] (-->/lib64/libc.so.6() [0x30daa47a00] (-->/usr/local/lib/libglusterfs.so.0(synctask_wrap+0x12) [0x7fa20b7c6ec2] (-->/usr/local/lib/glusterfs/3.7dev/xlator/cluster/distribute.so(dht_migrate_file+0x23f) [0x7fa200fdb58f]))) 0-dict: dict is NULL Change-Id: I4c93e4485293b35d86ba07df4d583d2758ec3f49 BUG: 1138395 Signed-off-by: Vijay Bellur <vbellur> Reviewed-on-master: http://review.gluster.org/8601 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Shyamsundar Ranganathan <srangana> Reviewed-on: http://review.gluster.org/8782
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html [2] http://supercolony.gluster.org/pipermail/gluster-users/
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report. glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html [2] http://supercolony.gluster.org/mailman/listinfo/gluster-users