Bug 1583733

Summary: Poor write performance on gluster-block
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sunil Kumar Acharya <sheggodu>
Component: replicateAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.3CC: amarts, amukherj, atumball, csaba, lbailey, mpillai, pkarampu, pmulay, ravishankar, rcyriac, rgowdapp, rhinduja, rhs-bugs, sanandpa, sankarshan, shberry, sheggodu, storage-qa-internal, xiubli
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.3.1 Async   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-54.12 Doc Type: Bug Fix
Doc Text:
Previously, eager-lock was disabled for volumes hosted by a block, because conflicting writes were handled incorrectly when eager-lock is enabled. Hence, the performance of gluster backed block devices was insufficient when eager-lock was enabled. This update fixes the eager-lock handling for conflicting writes. Thus, when eager-lock is enabled, performance of gluster backed block device is enhanced. To observe this performance improvement, the Gluster administrator needs to enable eager-lock on old block hosting volumes. Also, the eager-lock option is enabled by default for all new volumes.
Story Points: ---
Clone Of: 1491785 Environment:
Last Closed: 2018-07-19 06:00:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1491785, 1499644    
Bug Blocks:    

Comment 6 Pranith Kumar K 2018-06-25 13:16:12 UTC
gluster-block now uses the improved eager-lock implementation to reduce the number of network operations. To get this effect on the old block hosting volumes  we need to enable cluster.eager-lock option after all the gluster pods are upgraded to the latest release.
# gluster volume set <volname> cluster.eager-lock on

Comment 8 Nag Pavan Chilakam 2018-07-09 09:37:47 UTC
Hello Manoj,
Any update on this, as qe is targetting to move all 331-async bugs to verified by tomorrow EOD?

Comment 9 Manoj Pillai 2018-07-09 15:12:46 UTC
With build:
glusterfs-fuse-3.8.4-54.14.el7rhgs.x86_64
gluster-block-0.2.1-20.el7rhgs.x86_64
glusterfs-libs-3.8.4-54.14.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-54.14.el7rhgs.x86_64
glusterfs-api-3.8.4-54.14.el7rhgs.x86_64
glusterfs-server-3.8.4-54.14.el7rhgs.x86_64
libtcmu-1.2.0-20.el7rhgs.x86_64
tcmu-runner-1.2.0-20.el7rhgs.x86_64
glusterfs-3.8.4-54.14.el7rhgs.x86_64
glusterfs-cli-3.8.4-54.14.el7rhgs.x86_64

Repeating the random write tests on a new setup. Only 3 systems available with 10GbE, so co-locating the client on one of the servers.

For a similar fio test I see:
glusterfs-fuse: 10610 IOPS
gluster-block: 9216 IOPS

So that looks good.

Also not seeing the write-amplification that we were seeing when the performance was poor (bz #1480188):
sdm               0.00     0.00    0.00 10504.60     0.00 42018.40 [at initiator]
sdb               0.00     4.00    0.00 10517.80     0.00 44974.05 [at brick]

Comment 13 Pranith Kumar K 2018-07-17 05:55:45 UTC
Meaning given by first two sentences is a bit misleading.
It is: "Previously, eager-lock was disabled for volumes hosted by a block. Due to this reason, the conflicting writes were handled incorrectly"

But it is supposed to convey: "Previously, eager-lock was disabled for volumes hosted by a block because conflicting writes were handled incorrectly when eager-lock is enabled"

Rest of the doc-text looked okay.

Comment 15 Pranith Kumar K 2018-07-17 06:14:42 UTC
It looks good to me. We don't need to explicitly say that it is enabled by default for new volumes?

Comment 22 errata-xmlrpc 2018-07-19 06:00:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2222