Bug 1583733 - Poor write performance on gluster-block
Summary: Poor write performance on gluster-block
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: replicate
Version: rhgs-3.3
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: RHGS 3.3.1 Async
Assignee: Pranith Kumar K
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On: 1491785 1499644
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-29 14:49 UTC by Sunil Kumar Acharya
Modified: 2018-09-24 03:19 UTC (History)
19 users (show)

Fixed In Version: glusterfs-3.8.4-54.12
Doc Type: Bug Fix
Doc Text:
Previously, eager-lock was disabled for volumes hosted by a block, because conflicting writes were handled incorrectly when eager-lock is enabled. Hence, the performance of gluster backed block devices was insufficient when eager-lock was enabled. This update fixes the eager-lock handling for conflicting writes. Thus, when eager-lock is enabled, performance of gluster backed block device is enhanced. To observe this performance improvement, the Gluster administrator needs to enable eager-lock on old block hosting volumes. Also, the eager-lock option is enabled by default for all new volumes.
Clone Of: 1491785
Environment:
Last Closed: 2018-07-19 06:00:07 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2222 0 None None None 2018-07-19 06:01:41 UTC

Comment 6 Pranith Kumar K 2018-06-25 13:16:12 UTC
gluster-block now uses the improved eager-lock implementation to reduce the number of network operations. To get this effect on the old block hosting volumes  we need to enable cluster.eager-lock option after all the gluster pods are upgraded to the latest release.
# gluster volume set <volname> cluster.eager-lock on

Comment 8 Nag Pavan Chilakam 2018-07-09 09:37:47 UTC
Hello Manoj,
Any update on this, as qe is targetting to move all 331-async bugs to verified by tomorrow EOD?

Comment 9 Manoj Pillai 2018-07-09 15:12:46 UTC
With build:
glusterfs-fuse-3.8.4-54.14.el7rhgs.x86_64
gluster-block-0.2.1-20.el7rhgs.x86_64
glusterfs-libs-3.8.4-54.14.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-54.14.el7rhgs.x86_64
glusterfs-api-3.8.4-54.14.el7rhgs.x86_64
glusterfs-server-3.8.4-54.14.el7rhgs.x86_64
libtcmu-1.2.0-20.el7rhgs.x86_64
tcmu-runner-1.2.0-20.el7rhgs.x86_64
glusterfs-3.8.4-54.14.el7rhgs.x86_64
glusterfs-cli-3.8.4-54.14.el7rhgs.x86_64

Repeating the random write tests on a new setup. Only 3 systems available with 10GbE, so co-locating the client on one of the servers.

For a similar fio test I see:
glusterfs-fuse: 10610 IOPS
gluster-block: 9216 IOPS

So that looks good.

Also not seeing the write-amplification that we were seeing when the performance was poor (bz #1480188):
sdm               0.00     0.00    0.00 10504.60     0.00 42018.40 [at initiator]
sdb               0.00     4.00    0.00 10517.80     0.00 44974.05 [at brick]

Comment 13 Pranith Kumar K 2018-07-17 05:55:45 UTC
Meaning given by first two sentences is a bit misleading.
It is: "Previously, eager-lock was disabled for volumes hosted by a block. Due to this reason, the conflicting writes were handled incorrectly"

But it is supposed to convey: "Previously, eager-lock was disabled for volumes hosted by a block because conflicting writes were handled incorrectly when eager-lock is enabled"

Rest of the doc-text looked okay.

Comment 15 Pranith Kumar K 2018-07-17 06:14:42 UTC
It looks good to me. We don't need to explicitly say that it is enabled by default for new volumes?

Comment 22 errata-xmlrpc 2018-07-19 06:00:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2222


Note You need to log in before you can comment on or make changes to this bug.