Bug 1139995 - [DHT:REBALANCE]: Rebalance failures are seen with error message " remote operation failed: File exists"
Summary: [DHT:REBALANCE]: Rebalance failures are seen with error message " remote ope...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: 3.4.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On: 1110694 1116150
Blocks: glusterfs-3.4.6 1129541 1138385
TreeView+ depends on / blocked
 
Reported: 2014-09-10 07:04 UTC by Raghavendra G
Modified: 2015-04-13 07:08 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1116150
Environment:
Last Closed: 2015-04-13 07:08:31 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Comment 1 Anand Avati 2014-09-10 08:38:05 UTC
REVIEW: http://review.gluster.org/8674 (cluster/dht: Fix races to avoid deletion of linkto file) posted (#1) for review on release-3.4 by Raghavendra G (rgowdapp@redhat.com)

Comment 2 Anand Avati 2014-09-24 15:14:24 UTC
REVIEW: http://review.gluster.org/8674 (cluster/dht: Fix races to avoid deletion of linkto file) posted (#2) for review on release-3.4 by Raghavendra G (rgowdapp@redhat.com)

Comment 3 Anand Avati 2014-09-25 12:01:03 UTC
REVIEW: http://review.gluster.org/8674 (cluster/dht: Fix races to avoid deletion of linkto  file) posted (#3) for review on release-3.4 by Raghavendra G (rgowdapp@redhat.com)

Comment 4 Anand Avati 2014-09-25 13:53:53 UTC
REVIEW: http://review.gluster.org/8674 (cluster/dht: Fix races to avoid deletion of linkto  file) posted (#4) for review on release-3.4 by Raghavendra G (rgowdapp@redhat.com)

Comment 5 Anand Avati 2014-10-09 05:40:33 UTC
REVIEW: http://review.gluster.org/8674 (cluster/dht: Fix races to avoid deletion of linkto  file) posted (#5) for review on release-3.4 by Raghavendra G (rgowdapp@redhat.com)

Comment 6 Anand Avati 2014-10-09 07:34:54 UTC
REVIEW: http://review.gluster.org/8674 (cluster/dht: Fix races to avoid deletion of linkto  file) posted (#6) for review on release-3.4 by Raghavendra G (rgowdapp@redhat.com)

Comment 7 Anand Avati 2014-10-20 14:53:59 UTC
COMMIT: http://review.gluster.org/8674 committed in release-3.4 by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit b3387c83a0d968db7428920f46434f09f4922e53
Author: Venkatesh Somyajulu <vsomyaju@redhat.com>
Date:   Thu Sep 25 17:24:09 2014 +0530

    cluster/dht: Fix races to avoid deletion of linkto
     file
    
    Explanation of Race between rebalance processes:
    https://bugzilla.redhat.com/show_bug.cgi?id=1110694#c4
    
    scenario-1:
    ===========
    
    STATE 1:                          BRICK-1
    only one brick                   Cached File
    in the system
    
    STATE 2:
    Add brick-2                       BRICK-1                BRICK-2
    
    STATE 3:                                       Lookup of File on brick-2
                                                   by this node's rebalance
                                                   will fail because hashed
                                                   file is not created yet.
                                                   So dht_lookup_everywhere is
                                                   about to get called.
    
    STATE 4:                         As part of lookup
                                     link file at brick-2
                                     will be created.
    
    STATE 5:                         getxattr to check that
                                     cached file belongs to
                                     this node is done
    
    STATE 6:
    
                                                dht_lookup_everywhere_cbk detects
                                                the link created by rebalance-1.
                                                It will unlink it.
    
    STATE 7:                        getxattr at the link
                                    file with "pathinfo" key
                                    will be called will fail
                                    as the link file is deleted
                                    by rebalance on node-2
    
    Fix:
    So in the STATE 6, we should avoid the deletion of link file. Every time
    dht_lookup_everywhere gets called, lookup will be performed on all the nodes.
    So to avoid STATE 6, if linkto file is found, it is not deleted until valid
    case is found in dht_lookup_everywhere_done.
    
    Case 1: if linkto file points to cached node, and cached file exists,
            uwind with success.
    
    Case 2: if linkto does not point to current cached node, and cached file
            exists:
            a) Unlink stale link file
            b) Create new link file
    
    Case 3: Only linkto file exists:
            Delete linkto file
    
    Case 4: Only cached file
            Create link file (Handled event without patch)
    
    Case 5: Neither cached nor hashed file is present
            Return with ENOENT (handled even without patch)
    
    Reviewed-on: http://review.gluster.org/8231
    
    ******************************************************************************
    
    scenario-2:
    ===========
    cluster/dht: Modified logic of linkto file deletion on non-hashed
    
    Currently whenever dht_lookup_everywhere gets called, if in
    dht_lookup_everywhere_cbk, a linkto file is found on non-hashed
    subvolume, file is unlinked. But there are cases when this file
    is under migration. Under such condition, we should avoid deletion
    of file.
    
    When  some other rebalance process changes the layout of parent
    such that dst_file (w.r.t. migration) falls on non-hashed node,
    then may be lookup could have found it as linkto file but just
    before unlink, file  is under migration or already migrated
    In such cased unlink can be avoided.
    
    Race:
    -------
    If we have two bricks (brick-1 and brick-2) with initial file "a"
    under BaseDir which is hashed as well as cached on (brick-1).
    
    Assume "a"  hashing gives 44.
    
                                  Brick-1              Brick-2
    
    Initial Setup:               BaseDir/a             BaseDir
                                 [1-50]                [51-100]
    
    Now add new-brick Brick-3.
    
    1. Rebalance-1 on node Node-1 (Brick-1 node) will reset
    the BaseDir Layout.
    
    2. After that it will perform
    a)  Create linkto file on  new-hashed (brick-2)
    b)  Perform file migration.
    
    1.Rebalance-1 Fixes the base-layout:
                     Brick-1             Brick-2           Brick-3
                     ---------         ----------         ------------
                     BaseDir/a            BaseDir           BaseDir
                      [1-33]              [34-66]           [67-100]
    
    2. Only a) is     BaseDir/a          BaseDir/a(linkto)   BaseDir
       performed                         Create linktofile
    
    Now rebalance 2 on node-2 jumped in and it will perform
    step 1 and 2-a.
    
    After (rebal-2, step-1), it changes the layout of the BaseDir.
                        BaseDir/a     BaseDir/a(link)    BaseDir
                        [67-100]           [1-33]        [34-66]
    
    For  (rebale-2, step-2), It will perform lookup at Brick-3 as w.r.t new
    layout 44 falls for brick-3. But lookup will fail.
    So  dht_lookup_everywhere gets called.
    
    NOTE: On brick-2 by rebalance-1, a linkto file was created.
    
    Currently that linkto files gets deleted by rebalance-2 lookup as it
    is considered as stale linkto file.  But  with patch if rebalance is
    already in progress or rebalance is over,  linkto file will not be
    unlinked. If rebalance is in progress fd will be  open and if rebalance
    is over then linkto file wont be set.
    
    Reviewed-on: http://review.gluster.org/8345
    
    *******************************************************************************
    
    scenario-3:
    ===========
    
    cluster/dht: Added keys in dht_lookup_everywhere_done
    
    Case where both cached  (C1)  and hashed file are found,
    but hash does not point to above cached node (C1), then
    dont unlink if either fd-is-open on hashed or
    linkto-xattr is not found.
    
    Reviewed-on: http://review.gluster.org/8429
    BUG: 1139995
    Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
    Change-Id: I86d0a21d4c0501c45d837101ced4f96d6fedc5b9
    Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: susant palai <spalai@redhat.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>
    Reviewed-on: http://review.gluster.org/8674
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>


Note You need to log in before you can comment on or make changes to this bug.