Bug 1261716 - Sharding - read/write performance improvements for VM workload
Sharding - read/write performance improvements for VM workload
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: sharding (Show other bugs)
3.7.4
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Krutika Dhananjay
bugs@gluster.org
: Triaged
Depends On: 1258905
Blocks: Gluster-HC-1 glusterfs-3.7.5
  Show dependency treegraph
 
Reported: 2015-09-09 23:25 EDT by Krutika Dhananjay
Modified: 2015-12-01 11:45 EST (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1258905
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Krutika Dhananjay 2015-09-09 23:25:28 EDT
+++ This bug was initially created as a clone of Bug #1258905 +++

Description of problem:

Paul Cuzner in his testing of sharding in hyperconvergence environment noted a 3x write latency and 2x read latency.
One network operation that can be eliminated among the many things that shard translator does in every WRITEV and READV fop is the LOOKUP that is done on the zeroth shard to fetch the size and block_count xattr. Since VM workload is a single-writer use-case, and the client that wrote to a file is always the one that is going to read it, the size and block count xattr could be cached (and kept up-to-date) in the inode ctx of the main file, thereby saving the need for the (extra) LOOKUP every time.

The other place where network fops can be avoided is XATTROP, if/when there is a WRITEV that does not change the file size && block count,

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Krutika Dhananjay on 2015-09-01 21:57:23 EDT ---

(In reply to Krutika Dhananjay from comment #0)
> Description of problem:
> 
> Paul Cuzner in his testing of sharding in hyperconvergence environment noted
> a 3x write latency and 2x read latency.
> One network operation that can be eliminated among the many things that
> shard translator does in every WRITEV and READV fop is the LOOKUP that is
> done on the zeroth shard to fetch the size and block_count xattr. Since VM
> workload is a single-writer use-case, and the client that wrote to a file is
> always the one that is going to read it, the size and block count xattr
> could be cached (and kept up-to-date) in the inode ctx of the main file,
> thereby saving the need for the (extra) LOOKUP every time.
> 

Forgot to add that the credit for this idea above goes to Pranith Kumar K.

-Krutika

> The other place where network fops can be avoided is XATTROP, if/when there
> is a WRITEV that does not change the file size && block count,
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1.
> 2.
> 3.
> 
> Actual results:
> 
> 
> Expected results:
> 
> 
> Additional info:

--- Additional comment from Paul Cuzner on 2015-09-01 22:54:59 EDT ---

(In reply to Krutika Dhananjay from comment #0)
> Description of problem:
> 
> Paul Cuzner in his testing of sharding in hyperconvergence environment noted
> a 3x write latency and 2x read latency.
> One network operation that can be eliminated among the many things that
> shard translator does in every WRITEV and READV fop is the LOOKUP that is
> done on the zeroth shard to fetch the size and block_count xattr. Since VM
> workload is a single-writer use-case, and the client that wrote to a file is
> always the one that is going to read it, the size and block count xattr
> could be cached (and kept up-to-date) in the inode ctx of the main file,
> thereby saving the need for the (extra) LOOKUP every time.
> 
> The other place where network fops can be avoided is XATTROP, if/when there
> is a WRITEV that does not change the file size && block count,
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1.
> 2.
> 3.
> 
> Actual results:
> 
> 
> Expected results:
> 
> 
> Additional info:

This sounds great but I have to ask about shared vdisks - for example, RHEV supports vdisk sharing across vm's. Typically this would mean that the disk is only ever online to one vm at a time - but I wanted to make sure that this use case is considered.

--- Additional comment from Vijay Bellur on 2015-09-08 07:59:02 EDT ---

REVIEW: http://review.gluster.org/12126 (features/shard: Performance improvements in IO path) posted (#1) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-09 06:46:07 EDT ---

REVIEW: http://review.gluster.org/12126 (features/shard: Performance improvements in IO path) posted (#2) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-09 06:46:09 EDT ---

REVIEW: http://review.gluster.org/12138 (features/shard: Performance improvements in IO path - Part 2) posted (#1) for review on master by Krutika Dhananjay (kdhananj@redhat.com)
Comment 2 Pranith Kumar K 2015-10-14 06:28:17 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-glusterfs-3.7.5, please open a new bug report.

glusterfs-glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 3 Pranith Kumar K 2015-10-14 06:28:17 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-glusterfs-3.7.5, please open a new bug report.

glusterfs-glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 4 Pranith Kumar K 2015-10-14 06:37:44 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.5, please open a new bug report.

glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 5 Pranith Kumar K 2015-10-14 06:37:44 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.5, please open a new bug report.

glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.