REVIEW: http://review.gluster.org/8717 (cluster/dht: Fix races to avoid deletion of linkto file) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)
REVIEW: http://review.gluster.org/8719 (cluster/dht: Modified logic of linkto file deletion on non-hashed) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)
REVIEW: http://review.gluster.org/8720 (cluster/dht: Added keys in dht_lookup_everywhere_done) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)
REVIEW: http://review.gluster.org/8725 (storage/posix: Don't unlink .glusterfs-hardlink before linkto check) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)
REVIEW: http://review.gluster.org/8731 (cluster/dht: Added code to capture races in dht-lookup path) posted (#1) for review on release-3.5 by N Balachandran (nbalacha)
COMMIT: http://review.gluster.org/8717 committed in release-3.5 by Niels de Vos (ndevos) ------ commit 3f3e57cef2cd11f336a6d6496de37c89a3e90789 Author: Venkatesh Somyajulu <vsomyaju> Date: Wed Sep 10 22:18:34 2014 +0530 cluster/dht: Fix races to avoid deletion of linkto file Explanation of Race between rebalance processes: https://bugzilla.redhat.com/show_bug.cgi?id=1110694#c4 STATE 1: BRICK-1 only one brick Cached File in the system STATE 2: Add brick-2 BRICK-1 BRICK-2 STATE 3: Lookup of File on brick-2 by this node's rebalance will fail because hashed file is not created yet. So dht_lookup_everywhere is about to get called. STATE 4: As part of lookup link file at brick-2 will be created. STATE 5: getxattr to check that cached file belongs to this node is done STATE 6: dht_lookup_everywhere_cbk detects the link created by rebalance-1. It will unlink it. STATE 7: getxattr at the link file with "pathinfo" key will be called will fail as the link file is deleted by rebalance on node-2 Fix: So in the STATE 6, we should avoid the deletion of link file. Every time dht_lookup_everywhere gets called, lookup will be performed on all the nodes. So to avoid STATE 6, if linkto file is found, it is not deleted until valid case is found in dht_lookup_everywhere_done. Case 1: if linkto file points to cached node, and cached file exists, uwind with success. Case 2: if linkto does not point to current cached node, and cached file exists: a) Unlink stale link file b) Create new link file Case 3: Only linkto file exists: Delete linkto file Case 4: Only cached file Create link file (Handled event without patch) Case 5: Neither cached nor hashed file is present Return with ENOENT (handled even without patch) Change-Id: Ibf53671410d8d613b8e2e7e5d0ec30fc7dcc0298 BUG: 1129541 Signed-off-by: Venkatesh Somyajulu <vsomyaju> Reviewed-on: http://review.gluster.org/8231 Reviewed-by: Vijay Bellur <vbellur> Tested-by: Vijay Bellur <vbellur> (cherry picked from commit 74d92e322e3c9f4f70ddfbf9b0e2140922009658) Signed-off-by: Nithya Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/8717 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy>
COMMIT: http://review.gluster.org/8719 committed in release-3.5 by Niels de Vos (ndevos) ------ commit e24e812d25f10c008b80d72bc8b1fb2c401bd892 Author: Venkatesh Somyajulu <vsomyaju> Date: Wed Sep 10 22:52:43 2014 +0530 cluster/dht: Modified logic of linkto file deletion on non-hashed Currently whenever dht_lookup_everywhere gets called, if in dht_lookup_everywhere_cbk, a linkto file is found on non-hashed subvolume, file is unlinked. But there are cases when this file is under migration. Under such condition, we should avoid deletion of file. When some other rebalance process changes the layout of parent such that dst_file (w.r.t. migration) falls on non-hashed node, then may be lookup could have found it as linkto file but just before unlink, file is under migration or already migrated In such cased unlink can be avoided. Race: ------- If we have two bricks (brick-1 and brick-2) with initial file "a" under BaseDir which is hashed as well as cached on (brick-1). Assume "a" hashing gives 44. Brick-1 Brick-2 Initial Setup: BaseDir/a BaseDir [1-50] [51-100] Now add new-brick Brick-3. 1. Rebalance-1 on node Node-1 (Brick-1 node) will reset the BaseDir Layout. 2. After that it will perform a) Create linkto file on new-hashed (brick-2) b) Perform file migration. 1.Rebalance-1 Fixes the base-layout: Brick-1 Brick-2 Brick-3 --------- ---------- ------------ BaseDir/a BaseDir BaseDir [1-33] [34-66] [67-100] 2. Only a) is BaseDir/a BaseDir/a(linkto) BaseDir performed Create linktofile Now rebalance 2 on node-2 jumped in and it will perform step 1 and 2-a. After (rebal-2, step-1), it changes the layout of the BaseDir. BaseDir/a BaseDir/a(link) BaseDir [67-100] [1-33] [34-66] For (rebale-2, step-2), It will perform lookup at Brick-3 as w.r.t new layout 44 falls for brick-3. But lookup will fail. So dht_lookup_everywhere gets called. NOTE: On brick-2 by rebalance-1, a linkto file was created. Currently that linkto files gets deleted by rebalance-2 lookup as it is considered as stale linkto file. But with patch if rebalance is already in progress or rebalance is over, linkto file will not be unlinked. If rebalance is in progress fd will be open and if rebalance is over then linkto file wont be set. Change-Id: I3fee0d28de3c76197325536a9e30099d2413f079 BUG: 1129541 Signed-off-by: Venkatesh Somyajulu <vsomyaju> Reviewed-on: http://review.gluster.org/8345 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Reviewed-by: Shyamsundar Ranganathan <srangana> Reviewed-by: Vijay Bellur <vbellur> (cherry picked from commit 966997992bdbd5fffc632bf705678e287ed50bf7) Signed-off-by: Nithya Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/8719 Reviewed-by: Jeff Darcy <jdarcy>
COMMIT: http://review.gluster.org/8720 committed in release-3.5 by Niels de Vos (ndevos) ------ commit 6f015ab7771f0ab515b57b5333777f04a04167e6 Author: Venkatesh Somyajulu <vsomyaju> Date: Wed Sep 10 23:00:07 2014 +0530 cluster/dht: Added keys in dht_lookup_everywhere_done Case where both cached (C1) and hashed file are found, but hash does not point to above cached node (C1), then dont unlink if either fd-is-open on hashed or linkto-xattr is not found. Change-Id: I7ef49b88d2c88bf9d25d3aa7893714e6c0766c67 BUG: 1129541 Signed-off-by: Venkatesh Somyajulu <vsomyaju> Change-Id: I86d0a21d4c0501c45d837101ced4f96d6fedc5b9 Signed-off-by: Venkatesh Somyajulu <vsomyaju> Reviewed-on: http://review.gluster.org/8429 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: susant palai <spalai> Reviewed-by: Raghavendra G <rgowdapp> Reviewed-by: Vijay Bellur <vbellur> (cherry picked from commit 718f10e0d68715be2d73e677974629452485c699) Signed-off-by: Nithya Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/8720 Reviewed-by: Jeff Darcy <jdarcy>
REVIEW: http://review.gluster.org/8725 (storage/posix: Don't unlink .glusterfs-hardlink before linkto check) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)
REVIEW: http://review.gluster.org/8731 (cluster/dht: Added code to capture races in dht-lookup path) posted (#2) for review on release-3.5 by N Balachandran (nbalacha)
REVIEW: http://review.gluster.org/8725 (storage/posix: Don't unlink .glusterfs-hardlink before linkto check) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)
REVIEW: http://review.gluster.org/8731 (cluster/dht: Added code to capture races in dht-lookup path) posted (#3) for review on release-3.5 by N Balachandran (nbalacha)
COMMIT: http://review.gluster.org/8725 committed in release-3.5 by Niels de Vos (ndevos) ------ commit 98e8883787dc6943c4a3fc8282f91a8c73fb0803 Author: Venkatesh Somyajulu <vsomyaju> Date: Wed Sep 10 23:38:02 2014 +0530 storage/posix: Don't unlink .glusterfs-hardlink before linkto check BUG: 1129541 Change-Id: I90a10ac54123fbd8c7383ddcbd04e8879ae51232 Signed-off-by: Venkatesh Somyajulu <vsomyaju> Reviewed-on: http://review.gluster.org/8559 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: N Balachandran <nbalacha> Reviewed-by: Vijay Bellur <vbellur> (cherry picked from commit b23be2e7581c6aa295053dc8866cab841ae374b6) Signed-off-by: Nithya Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/8725 Reviewed-by: Niels de Vos <ndevos>
COMMIT: http://review.gluster.org/8731 committed in release-3.5 by Niels de Vos (ndevos) ------ commit 0a4119e321be46bf274e1ee4251a15609ad09df7 Author: Venkatesh Somyajulu <vsomyaju> Date: Thu Sep 11 14:18:32 2014 +0530 cluster/dht: Added code to capture races in dht-lookup path Change-Id: I9270d2d40ebd4b113ff961583dfda7754741f15b BUG: 1129541 Signed-off-by: Venkatesh Somyajulu <vsomyaju> Reviewed-on: http://review.gluster.org/8430 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur> (cherry picked from commit bb2d5f49b5684e6484af16a580870cfe104aecd2) Signed-off-by: Nithya Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/8731 Reviewed-by: Niels de Vos <ndevos>
The first (and last?) Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-October/018990.html [2] http://supercolony.gluster.org/pipermail/gluster-users/
The second Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions have been made available on [2] to make testing easier. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019359.html [2] http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.3beta2/
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.3, please reopen this bug report. glusterfs-3.5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/announce/2014-November/000042.html [2] http://supercolony.gluster.org/pipermail/gluster-users/