Bug 1512460

Summary: disperse eager-lock degrades performance for file create workloads
Product: [Community] GlusterFS Reporter: Ashish Pandey <aspandey>
Component: disperseAssignee: bugs <bugs>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.13CC: bugs, jahernan, mpillai, pkarampu
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.13.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1502610 Environment:
Last Closed: 2017-12-08 17:46:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1502610, 1530519    
Bug Blocks: 1502455    

Description Ashish Pandey 2017-11-13 09:49:45 UTC 
+++ This bug was initially created as a clone of Bug #1502610 +++

+++ This bug was initially created as a clone of Bug #1502455 +++

Description of problem:
The current behavior of the option disperse.eager-lock is not optimal:

disperse.eager-lock on: good performance on large-file read/write, but actually degrades performance for many file create workloads. The degradation for file create workloads seems to be due to lock contention on directories.

disperse.eager-lock off: loses the performance advantages of eager-locking for large-file access, but better performance on file create workloads than with disperse.eager-lock on.

We should fix eager locking so that it can be kept on without incurring a performance penalty on file create workloads.

Version-Release number of selected component (if applicable):
glusterfs*-3.12.1-2.el7.x86_64

How reproducible:
Consistently

--- Additional comment from Manoj Pillai on 2017-10-16 07:32:52 CEST ---


IMO, a solution that adds a separate option to control eager locking in the case of directories would be acceptable, and probably simple. 

So the default could be:
disperse.dir-eager-lock off: applies to directories
disperse.eager-lock on: applies to regular files

Would that work?

--- Additional comment from Xavier Hernandez on 2017-10-16 10:12:26 CEST ---

I guess this problem happens when multiple clients are creating files on the same directory, right ? otherwise, eager-locking shouldn't interfere with file creation (in fact it should be faster).

In cases where multiple clients access the same directory, then yes, we could keep a separate configuration for this purpose. However, is it really necessary to have it disabled by default ? I think that an scenario where multiple clients are writing to the same directory is less probable than one where all writes to a single directory come from the same client.

--- Additional comment from Worker Ant on 2017-10-16 08:00:22 EDT ---

REVIEW: https://review.gluster.org/18530 (cluster/ec: create eager-lock option for non-regular files) posted (#1) for review on master by Xavier Hernandez (jahernan)

--- Additional comment from Worker Ant on 2017-10-16 13:31:01 EDT ---

REVIEW: https://review.gluster.org/18530 (cluster/ec: create eager-lock option for non-regular files) posted (#2) for review on master by Xavier Hernandez (jahernan)

--- Additional comment from Worker Ant on 2017-11-04 23:42:21 EDT ---

COMMIT: https://review.gluster.org/18530 committed in master by  

------------- cluster/ec: create eager-lock option for non-regular files

A new option is added to allow independent configuration of eager
locking for regular files and non-regular files.

Change-Id: I8f80e46d36d8551011132b15c0fac549b7fb1c60
BUG: 1502610
Signed-off-by: Xavier Hernandez <jahernan>
Comment 1 Worker Ant 2017-11-13 10:04:47 UTC 
REVIEW: https://review.gluster.org/18729 (cluster/ec: create eager-lock option for non-regular files) posted (#1) for review on release-3.13 by Ashish Pandey
Comment 2 Worker Ant 2017-11-14 17:42:13 UTC 
REVIEW: https://review.gluster.org/18779 (cluster/ec: Fix op-version for disperse.other-eager-lock) posted (#1) for review on release-3.13 by Xavier Hernandez
Comment 3 Worker Ant 2017-11-16 14:49:12 UTC 
COMMIT: https://review.gluster.org/18729 committed in release-3.13 by \"Ashish Pandey\" <aspandey> with a commit message- cluster/ec: create eager-lock option for non-regular files

A new option is added to allow independent configuration of eager
locking for regular files and non-regular files.

>Change-Id: I8f80e46d36d8551011132b15c0fac549b7fb1c60
>BUG: 1502610
>Signed-off-by: Xavier Hernandez <jahernan>

Change-Id: I8f80e46d36d8551011132b15c0fac549b7fb1c60
BUG: 1512460
Signed-off-by: Ashish Pandey <aspandey>
Comment 4 Worker Ant 2017-11-16 14:49:30 UTC 
COMMIT: https://review.gluster.org/18779 committed in release-3.13 by \"Xavier Hernandez\" <jahernan> with a commit message- cluster/ec: Fix op-version for disperse.other-eager-lock

The op-version used for the new option was wrong. It has been set
to 3.13.0.

>Change-Id: I88fbd7834e4a8018c8906303e734c251e90be8cf
>BUG: 1502610
>Signed-off-by: Xavier Hernandez <jahernan>

Change-Id: I88fbd7834e4a8018c8906303e734c251e90be8cf
BUG: 1512460
Signed-off-by: Xavier Hernandez <jahernan>
Comment 5 Shyamsundar 2017-12-08 17:46:04 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.13.0, please open a new bug report.

glusterfs-3.13.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-December/000087.html
[2] https://www.gluster.org/pipermail/gluster-users/