Bug 1385226

Summary:	arbiter volume write performance is bad with sharding
Product:	[Community] GlusterFS	Reporter:	Ravishankar N <ravishankar>
Component:	arbiter	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.7.15	CC:	bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.17	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1384906	Environment:
Last Closed:	2016-11-16 10:51:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1384906
Bug Blocks:	1375125, 1385224

Description Ravishankar N 2016-10-15 12:26:43 UTC

+++ This bug was initially created as a clone of Bug #1384906 +++

Description of problem:
Max Raba reported  that arbiter volume write performance is bad with sharding. See BZ 1375125

--- Additional comment from Worker Ant on 2016-10-14 06:58:54 EDT ---

REVIEW: http://review.gluster.org/15641 (afr: Take full locks in arbiter only for data transactions) posted (#1) for review on master by Ravishankar N (ravishankar)

--- Additional comment from Worker Ant on 2016-10-14 13:18:40 EDT ---

COMMIT: http://review.gluster.org/15641 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 3a97486d7f9d0db51abcb13dcd3bc9db935e3a60
Author: Ravishankar N <ravishankar>
Date:   Fri Oct 14 16:09:08 2016 +0530

    afr: Take full locks in arbiter only for data transactions
    
    Problem:
    Sharding exposed a bug in arbiter config. where `dd` throughput was
    extremely slow. Shard xlator was sending a fxattrop to update the file
    size immediately after a writev. Arbiter was incorrectly over-riding the
    LLONGMAX-1 start offset (for metadata domain locks) for this fxattrop,
    causing the inodelk to be taken on the data domain. And since the
    preceeding writev hadn't released the lock (afr does a 'lazy'
    unlock if write succeeds on all bricks), this degraded to a blocking
    lock causing extra lock/unlock calls and delays.
    
    Fix:
    Modify flock.l_len and flock.l_start to take full locks only for data
    transactions.
    
    Change-Id: I906895da2f2d16813607e6c906cb4defb21d7c3b
    BUG: 1384906
    Signed-off-by: Ravishankar N <ravishankar>
    Reported-by: Max Raba <max.raba>
    Reviewed-on: http://review.gluster.org/15641
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 1 Worker Ant 2016-10-15 12:28:11 UTC

REVIEW: http://review.gluster.org/15649 (afr: Take full locks in arbiter only for data transactions) posted (#1) for review on release-3.7 by Ravishankar N (ravishankar)

Comment 2 Worker Ant 2016-10-16 06:11:17 UTC

COMMIT: http://review.gluster.org/15649 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit 7f8916364ff2092e0391b294051dc76764ff02fc
Author: Ravishankar N <ravishankar>
Date:   Fri Oct 14 16:09:08 2016 +0530

    afr: Take full locks in arbiter only for data transactions
    
    Problem:
    Sharding exposed a bug in arbiter config. where `dd` throughput was
    extremely slow. Shard xlator was sending a fxattrop to update the file
    size immediately after a writev. Arbiter was incorrectly over-riding the
    LLONGMAX-1 start offset (for metadata domain locks) for this fxattrop,
    causing the inodelk to be taken on the data domain. And since the
    preceeding writev hadn't released the lock (afr does a 'lazy'
    unlock if write succeeds on all bricks), this degraded to a blocking
    lock causing extra lock/unlock calls and delays.
    
    Fix:
    Modify flock.l_len and flock.l_start to take full locks only for data
    transactions.
    
    > Reviewed-on: http://review.gluster.org/15641
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    (cherry picked from commit 3a97486d7f9d0db51abcb13dcd3bc9db935e3a60)
    
    Change-Id: I906895da2f2d16813607e6c906cb4defb21d7c3b
    BUG: 1385226
    Signed-off-by: Ravishankar N <ravishankar>
    Reported-by: Max Raba <max.raba>
    Reviewed-on: http://review.gluster.org/15649
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 3 Samikshan Bairagya 2016-11-16 10:51:48 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.17, please open a new bug report.

glusterfs-3.7.17 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-devel/2016-November/051414.html
[2] https://www.gluster.org/pipermail/gluster-users/