Bug 1149727

REVIEW: http://review.gluster.org/8946 (ec: Fix self-heal issues) posted (#1) for review on release-3.6 by Xavier Hernandez (xhernandez)

REVIEW: http://review.gluster.org/8946 (ec: Fix self-heal issues) posted (#2) for review on release-3.6 by Xavier Hernandez (xhernandez)

REVIEW: http://review.gluster.org/8946 (ec: Fix self-heal issues) posted (#3) for review on release-3.6 by Xavier Hernandez (xhernandez)

COMMIT: http://review.gluster.org/8946 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit 4522acc20bdd1ca17c053969ef7edce1bb6ede76
Author: Xavier Hernandez <xhernandez>
Date:   Wed Oct 8 09:20:11 2014 +0200

    ec: Fix self-heal issues
    
    Problem: Doing an 'ls' of a directory that has been modified while one
             of the bricks was down, sometimes returns the old directory
             contents.
    
    Cause: Directories are not marked when they are modified as files are.
           The ec xlator balances requests amongst available and healthy
           bricks. Since there is no way to detect that a directory is
           out of date in one of the bricks, it is used from time to time
           to return the directory contents.
    
    Solution: Basically the solution consists in use versioning information
              also for directories, however some additional changes have
              been necessary.
    
    Changes:
    
     * Use directory versioning:
    
         This required to lock full directory instead of a single entry for
         all requests that add or remove entries from it. This is needed to
         allow atomic version update. This affects the following fops:
    
             create, mkdir, mknod, link, symlink, rename, unlink, rmdir
    
         Another side effect is that opendir requires to do a previous
         lookup to get versioning information and discard out of date
         bricks for subsequent readdir(p) calls.
    
     * Restrict directory self-heal:
    
         Till now, when one discrepancy was found in lookup, a self-heal
         was automatically started. This caused the versioning information
         of a bad directory to be healed instantly, making the original
         problem to reapear again.
    
         To solve this, when a missing directory is detected in one or more
         bricks on lookup or opendir fops, only a partial self-heal is
         performed on it. A partial self-heal basically creates the
         directory but does not restore any additional information.
    
         This avoids that an 'ls' could repair the directory and cause the
         problem to happen again. With this change, output of 'ls' is
         always consistent. However, since the directory has been created
         in the brick, this allows any other operation on it (create new
         files, for example) to succeed on all bricks and not add additional
         work to the self-heal process.
    
         To force a self-heal of a directory, any other operation must be
         done on it. For example a getxattr.
    
         With these changes, the correct healing procedure that would avoid
         inconsistent directory browsing consists on a post-order traversal
         of directoriesi being healed. This way, the directory contents will
         be healed before healing the directory itslef.
    
     * Additional changes to fix self-heal errors
    
         - Don't use fop->fd to decide between fd/loc.
    
             open, opendir and create have an fd, but the correct data is in
             loc.
    
         - Fix incorrect management of bad bricks per inode/fd.
    
         - Fix incorrect selection of fop's target bricks when there are bad
           bricks involved.
    
         - Improved ec_loc_parent() to always return a parent loc as
           complete as possible.
    
    This is a backport of http://review.gluster.org/8916/
    
    Change-Id: Iaf3df174d7857da57d4a87b4a8740a7048b366ad
    BUG: 1149727
    Signed-off-by: Xavier Hernandez <xhernandez>
    Reviewed-on: http://review.gluster.org/8946
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Gluster Build System <jenkins.com>

*** Bug 1159485 has been marked as a duplicate of this bug. ***

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users