Bug 1095888 - dist-geo-rep: while doing rm -rf on master mount, slave glusterfs crashed in io-cache.
Summary: dist-geo-rep: while doing rm -rf on master mount, slave glusterfs crashed in ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Raghavendra G
QA Contact:
URL:
Whiteboard:
Depends On: 1095671 1126369
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-08 18:25 UTC by Raghavendra G
Modified: 2014-11-11 08:32 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.6.0beta1
Clone Of: 1095671
Environment:
Last Closed: 2014-11-11 08:32:09 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Comment 1 Anand Avati 2014-05-08 18:38:36 UTC
REVIEW: http://review.gluster.org/7715 (cluster/dht: fix hole calculation when lookup returns errors from all subvolumes) posted (#1) for review on master by Raghavendra G (rgowdapp)

Comment 2 Anand Avati 2014-05-08 18:42:50 UTC
REVIEW: http://review.gluster.org/7715 (cluster/dht: fix hole calculation when lookup returns errors from all subvolumes) posted (#2) for review on master by Raghavendra G (rgowdapp)

Comment 3 Krutika Dhananjay 2014-05-28 07:01:22 UTC
http://review.gluster.org/7715

Comment 4 Anand Avati 2014-06-02 05:24:18 UTC
REVIEW: http://review.gluster.org/7941 (cluster/dht: fix hole calculation when lookup returns errors from all subvolumes) posted (#1) for review on master by Raghavendra G (rgowdapp)

Comment 5 Anand Avati 2014-06-02 05:24:25 UTC
REVIEW: http://review.gluster.org/7942 (cluster/dht: heal only when lookup is successful on atleast one subvolume.) posted (#1) for review on master by Raghavendra G (rgowdapp)

Comment 6 Anand Avati 2014-06-05 07:45:37 UTC
REVIEW: http://review.gluster.org/7493 (cluster/dht: Do layout self healing of directory for nameless lookup) posted (#6) for review on master by venkatesh somyajulu (vsomyaju)

Comment 7 Anand Avati 2014-06-12 12:44:02 UTC
REVIEW: http://review.gluster.org/7493 (cluster/dht: Do layout self healing of directory for nameless lookup) posted (#7) for review on master by venkatesh somyajulu (vsomyaju)

Comment 8 Anand Avati 2014-06-12 12:44:08 UTC
REVIEW: http://review.gluster.org/8049 (cluster/dht:  Bring option to opt gfid or name based hashing) posted (#1) for review on master by venkatesh somyajulu (vsomyaju)

Comment 9 Anand Avati 2014-06-12 18:00:27 UTC
REVIEW: http://review.gluster.org/8049 (cluster/dht:  Bring option to opt gfid or name based hashing) posted (#2) for review on master by Raghavendra G (rgowdapp)

Comment 10 Anand Avati 2014-06-12 18:00:34 UTC
REVIEW: http://review.gluster.org/7493 (cluster/dht: Do layout self healing of directory for nameless lookup) posted (#8) for review on master by Raghavendra G (rgowdapp)

Comment 11 Anand Avati 2014-06-16 17:55:18 UTC
REVIEW: http://review.gluster.org/8049 (cluster/dht:  Bring option to opt gfid or name based hashing) posted (#3) for review on master by Raghavendra G (rgowdapp)

Comment 12 Anand Avati 2014-06-16 18:35:57 UTC
REVIEW: http://review.gluster.org/7493 (cluster/dht: Do layout self healing of directory for nameless lookup) posted (#9) for review on master by venkatesh somyajulu (vsomyaju)

Comment 13 Anand Avati 2014-06-16 18:47:02 UTC
REVIEW: http://review.gluster.org/7493 (cluster/dht: Do layout self healing of directory for nameless lookup) posted (#10) for review on master by venkatesh somyajulu (vsomyaju)

Comment 14 Anand Avati 2014-06-17 06:38:53 UTC
REVIEW: http://review.gluster.org/8049 (cluster/dht:  Bring option to opt gfid or name based hashing) posted (#4) for review on master by venkatesh somyajulu (vsomyaju)

Comment 15 Anand Avati 2014-06-17 07:03:56 UTC
REVIEW: http://review.gluster.org/8049 (cluster/dht:  Bring option to choose gfid or name based hashing) posted (#5) for review on master by venkatesh somyajulu (vsomyaju)

Comment 16 Anand Avati 2014-06-17 07:04:03 UTC
REVIEW: http://review.gluster.org/7493 (cluster/dht: Do layout self healing of directory for nameless lookup) posted (#11) for review on master by venkatesh somyajulu (vsomyaju)

Comment 17 Anand Avati 2014-06-17 09:41:31 UTC
REVIEW: http://review.gluster.org/8049 (cluster/dht:  Bring option to choose gfid or name based hashing) posted (#6) for review on master by venkatesh somyajulu (vsomyaju)

Comment 18 Anand Avati 2014-06-17 09:41:43 UTC
REVIEW: http://review.gluster.org/7493 (cluster/dht: Do layout self healing of directory for nameless lookup) posted (#12) for review on master by venkatesh somyajulu (vsomyaju)

Comment 19 Anand Avati 2014-06-17 12:28:29 UTC
COMMIT: http://review.gluster.org/8049 committed in master by Vijay Bellur (vbellur) 
------
commit 3a499d170de2c7df06b127b709d27c64cef98886
Author: Venkatesh Somyajulu <vsomyaju>
Date:   Tue Jun 17 14:45:44 2014 +0530

    cluster/dht:  Bring option to choose gfid or name based hashing
    
    Change-Id: I11794eb2adceb88e75864aede450e904431a6273
    BUG: 1095888
    Signed-off-by: Venkatesh Somyajulu <vsomyaju>
    Reviewed-on: http://review.gluster.org/8049
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 20 Anand Avati 2014-06-17 12:39:34 UTC
COMMIT: http://review.gluster.org/7493 committed in master by Vijay Bellur (vbellur) 
------
commit 93832829016a0a51a8938c0c89c6bd09b3229c9f
Author: Venkatesh Somyajulu <vsomyaju>
Date:   Tue Jun 17 15:09:50 2014 +0530

    cluster/dht: Do layout self healing of directory for nameless lookup
    
    Problem: Currently in the  nameless lookup code path, if at the
             end of the lookup, even if it detects that layout
             anamolies are there, layout healing will not be done as
             there is no code to heal it.
             So there can be race between mkdir and lookup.
    
             Assume mkdir is going on from some other mount point,
             Say, M1. Directories are created on some nodes but layout
             is not set yet.
    
             Now from M2, nameless lookup goes, lookup will be success
             full as the directory is present on some of the nodes, but
             it won't heal layout. Now if create goes after lookup fop,
             because layout is absent, file creation will fail.
    
    Fix:     Included the code of layout self-heal in the nameless
             lookup path. At the end of lookup, layout will be computed
             as it would have been in the named lookup, but it will be
             set to those node only, where directory is present.
             So after that if create fop goes, the probabiliy to get the
             subvolume with proper hash-range is high now, so reduces
             the race window.
    
    Other:  Whenever a directory is created, we have to choose a brick
            from which we start allocating layout in a circular fashion.
            To calculate this starting brick, I have changed the candidate
            from name of the directory to gfid of the directory
    
            But to compute where a given file belongs, we will still
            use the name of the file. Hash computed from the name of the
            file should belong to any one of the directory-hash-range
    
            Calculation of hash for a file is acting as a consumer and the
            setting of directory layout based on gfid is acting as a producer,
            which are independent from each other.
    
    Change-Id: I3808c55082cd1b5c72d2c77cbbc063f55aa38bee
    BUG: 1095888
    Signed-off-by: Venkatesh Somyajulu <vsomyaju>
    Reviewed-on: http://review.gluster.org/7493
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra Bhat <raghavendra>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 21 Anand Avati 2014-06-18 12:20:42 UTC
REVIEW: http://review.gluster.org/8100 (cluster/dht: changed option name to hash-range-gfid) posted (#1) for review on master by venkatesh somyajulu (vsomyaju)

Comment 22 Anand Avati 2014-06-19 05:46:09 UTC
REVIEW: http://review.gluster.org/8100 (cluster/dht: changed option name to hash-range-gfid) posted (#2) for review on master by venkatesh somyajulu (vsomyaju)

Comment 23 Anand Avati 2014-06-19 05:46:52 UTC
REVIEW: http://review.gluster.org/8100 (cluster/dht: changed option name to layout-hash-gfid) posted (#3) for review on master by venkatesh somyajulu (vsomyaju)

Comment 24 Niels de Vos 2014-09-22 12:40:00 UTC
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 25 Niels de Vos 2014-11-11 08:32:09 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users


Note You need to log in before you can comment on or make changes to this bug.