Bug 1369077

Created attachment 1192926 [details]
server1 43.223

Created attachment 1192927 [details]
server2 42.48

Created attachment 1192928 [details]
Server3 43.142

Created attachment 1192929 [details]
server 4 client

Changing the component to replicate as it occurs on distribute replicate also. (Karan, feel free to correct me if I am wrong). Also assigning it to Pranith as he said he'd work on the fix:

Relevant technical discussions on IRC:
<itisravi>        pranithk1: are you free to talk about the bug Karan raised?
<itisravi>        its a day one issue IMO and not specific to afr.
<itisravi>        s/afr/arbiter   
<pranithk1>       itisravi: He said the bug is not recreatable in 3-way replication?
<itisravi>        pranithk1: It is..I've requested him to check again.
<itisravi>        pranithk1: so if mkdir fails on one replica subvol due to quorum not met etc , dht has no roll back
<itisravi>        thats the issue.
<pranithk1>       itisravi: Does it happen on plain replicate?
<itisravi>        pranithk1: no   
<itisravi>        pranithk1: its dht renamedir thing..
<pranithk1>       itisravi: okay, assign the bug to DHT giving the reason
<itisravi>        pranithk1: nithya was saying  if afr_inodelk can also have quorum checks, then renamedir will not happen
<itisravi>        so we will be good.
<itisravi>        instead of partially creating it on the up subvols of DHT 
<pranithk1>       itisravi: That is not a bad idea, send out a patch. Please tell her it only prevents the odds, won't fix the problem completely
<itisravi>        pranithk1: we can do it for afr_entrylk also then no?
<pranithk1>       itisravi: Actually the inodelk/finodelk needs to be reworked. I will send the patch
<pranithk1>       itisravi: yeah, that too
<itisravi>        pranithk1: I see , okay.

All 3.8.x bugs are now reported against version 3.8 (without .x). For more information, see http://www.gluster.org/pipermail/gluster-devel/2016-September/050859.html

REVIEW: http://review.gluster.org/15802 (cluster/afr: Fix bugs in [f]inodelk/[f]entrylk) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

REVIEW: http://review.gluster.org/15802 (cluster/afr: Fix bugs in [f]inodelk/[f]entrylk) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

COMMIT: http://review.gluster.org/15802 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 6be7bd936eb30aa8d2b908061f60e1534e797657
Author: Pranith Kumar K <pkarampu>
Date:   Mon Nov 7 14:47:34 2016 +0530

    cluster/afr: Fix bugs in [f]inodelk/[f]entrylk
    
    Problems:
    1) Inodelk is not taking quorum into account
    2) finodelk, [f]entrylk are not implemented correctly
    3) By default afr doesn't go for non-blocking parallel locks.
    
    Fix:
    Implemented a common framework which can be used by
    [f]inodelk/[f]entrylk.  Used quorum for the same.
    
    Change-Id: I239f13875a065298630d266941df10cfa3addc85
    BUG: 1369077
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/15802
    Tested-by: Krutika Dhananjay <kdhananj>
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Ravishankar N <ravishankar>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>

REVIEW: http://review.gluster.org/15984 (cluster/afr: Serialize conflicting locks on all subvols) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

REVIEW: http://review.gluster.org/15984 (cluster/afr: Serialize conflicting locks on all subvols) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

REVIEW: http://review.gluster.org/15984 (cluster/afr: Serialize conflicting locks on all subvols) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

REVIEW: http://review.gluster.org/15984 (cluster/afr: Serialize conflicting locks on all subvols) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

REVIEW: http://review.gluster.org/15984 (cluster/afr: Serialize conflicting locks on all subvols) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

REVIEW: http://review.gluster.org/16044 (tests: test parallel rmdirs to be successful) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

COMMIT: http://review.gluster.org/15984 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit a7d7ed90c9272a42168a91f92754d3a4be605da5
Author: Pranith Kumar K <pkarampu>
Date:   Thu Dec 1 09:42:19 2016 +0530

    cluster/afr: Serialize conflicting locks on all subvols
    
    Problem:
    1) When a blocking lock is issued and the parallel lock phase fails
    on all subvolumes with EAGAIN, it is not switching to serialized
    locking phase.
    2) When quorum is enabled and locks fail partially it is better
    to give errno returned by brick rather than the default
    quorum errno.
    
    Fix:
    Handled this error case and changed op_errno to reflect the actual
    errno in case of quorum error.
    
    BUG: 1369077
    Change-Id: Ifac2e4a13686e9fde601873012700966d56a7f31
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/15984
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Ravishankar N <ravishankar>

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/