Bug 1476665 - [Perf] : Large file sequential reads are off target by ~38% on FUSE/Ganesha
[Perf] : Large file sequential reads are off target by ~38% on FUSE/Ganesha
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
mainline
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Nithya Balachandran
:
Depends On:
Blocks: 1367266 1475136 1479303
  Show dependency treegraph
 
Reported: 2017-07-31 02:48 EDT by Nithya Balachandran
Modified: 2017-12-08 12:36 EST (History)
15 users (show)

See Also:
Fixed In Version: glusterfs-3.13.0
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1475136
: 1479303 (view as bug list)
Environment:
Last Closed: 2017-12-08 12:36:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nithya Balachandran 2017-07-31 02:48:27 EDT
+++ This bug was initially created as a clone of Bug #1475136 +++

Description of problem:
-----------------------

A regression seems to have been introduced in recent bits on large file seq reads :

3.2 : 2480044.05 kB/sec	
3.8.4-35 : 1538178.2 kB/sec


Regression : ~38%

This is on a vanilla volume,without PR,NL,mdcache.

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

3.8.4-35

How reproducible:
-----------------

100%

Additional info:
----------------
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 4b52bfb8-28fd-4e0f-8ee0-eb8116a296c4
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: server1:/bricks/testvol_brick0
Brick2: server2:/bricks/testvol_brick1
Brick3: server3:/bricks/testvol_brick2
Brick4: server4:/bricks/testvol_brick3
Options Reconfigured:
server.allow-insecure: on
performance.stat-prefetch: off
transport.address-family: inet
nfs.disable: on
Comment 1 Nithya Balachandran 2017-07-31 02:58:32 EDT
The changes made in https://review.gluster.org/#/c/17630/ have introduced a perf regression. 


The code has been changed to check if the fd is open on the dst subvol only if the FOP returns with EBADFD.
Comment 2 Worker Ant 2017-07-31 03:06:04 EDT
REVIEW: https://review.gluster.org/17922 (cluster/dht: Check for open fd only on EBADF) posted (#1) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 3 Worker Ant 2017-07-31 04:10:16 EDT
REVIEW: https://review.gluster.org/17922 (cluster/dht: Check for open fd only on EBADF) posted (#2) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 4 Worker Ant 2017-07-31 10:00:25 EDT
COMMIT: https://review.gluster.org/17922 committed in master by Raghavendra G (rgowdapp@redhat.com) 
------
commit 91c9f4a19fde4894576b398252c77f730832a26a
Author: N Balachandran <nbalacha@redhat.com>
Date:   Mon Jul 31 12:32:59 2017 +0530

    cluster/dht: Check for open fd only on EBADF
    
    DHT fd based fops will now check if the fd is
    open on the cached subvol only if the call fails
    with EBADF.
    
    This will improve performance for scenarios where
    a rebalance is not running which would be most of
    the time.
    
    Change-Id: Idfaeb8927af769c6110d07a165a0fe2307369239
    BUG: 1476665
    Signed-off-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-on: https://review.gluster.org/17922
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Comment 5 Worker Ant 2017-08-03 23:26:26 EDT
REVIEW: https://review.gluster.org/17970 (Revert "cluster/dht: Check for open fd only on EBADF") posted (#1) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 6 Worker Ant 2017-08-04 04:56:25 EDT
COMMIT: https://review.gluster.org/17970 committed in master by N Balachandran (nbalacha@redhat.com) 
------
commit 6d1068ddb35be19df36210c9fcaa7ce97e2a376a
Author: N Balachandran <nbalacha@redhat.com>
Date:   Fri Aug 4 08:52:31 2017 +0530

    Revert "cluster/dht: Check for open fd only on EBADF"
    
    This reverts commit 91c9f4a19fde4894576b398252c77f730832a26a.
    This patch needs to be reworked.
    
    Change-Id: I4c24f647c2b1abc68fc4e9fe6eb810418e2033aa
    BUG: 1476665
    Signed-off-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-on: https://review.gluster.org/17970
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Comment 7 Worker Ant 2017-08-04 05:21:40 EDT
REVIEW: https://review.gluster.org/17976 (cluster/dht: Check for open fd only on EBADF) posted (#1) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 8 Worker Ant 2017-08-04 05:35:31 EDT
REVIEW: https://review.gluster.org/17976 (cluster/dht: Check for open fd only on EBADF) posted (#2) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 9 Worker Ant 2017-08-08 06:21:28 EDT
COMMIT: https://review.gluster.org/17976 committed in master by Raghavendra G (rgowdapp@redhat.com) 
------
commit cdca1cb26a0aba390c6d8485c0d6d95e22ffc8bd
Author: N Balachandran <nbalacha@redhat.com>
Date:   Fri Aug 4 14:46:38 2017 +0530

    cluster/dht: Check for open fd only on EBADF
    
    DHT fd based fops used to check if the fd was open
    on the cached subvol before winding the call. However,
    this introduced a performance regression of about
    30% for reads.
    
    This check was introduced to handle cases where files
    were migrated while IOs were happening. As this is not
    the common case, dht will now check if the fd is
    open on the cached subvol only if the call fails
    with EBADF.
    
    This will prevent a performance hit where a rebalance
    is not running.
    
    Change-Id: I2035a858d63c3fcd22bb634055bbb0ad01686808
    BUG: 1476665
    Signed-off-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-on: https://review.gluster.org/17976
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Amar Tumballi <amarts@redhat.com>
    Reviewed-by: Susant Palai <spalai@redhat.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Comment 10 Nithya Balachandran 2017-08-08 11:13:03 EDT
One more patch required.
Comment 11 Worker Ant 2017-08-08 13:11:22 EDT
REVIEW: https://review.gluster.org/17999 (cluster/dht: EBADF handling for fremovexattr and fsetxattr) posted (#1) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 12 Worker Ant 2017-08-08 22:44:50 EDT
COMMIT: https://review.gluster.org/17999 committed in master by Raghavendra G (rgowdapp@redhat.com) 
------
commit 747a08d34e2a1e94d7fce68a3577370288bb1955
Author: N Balachandran <nbalacha@redhat.com>
Date:   Tue Aug 8 22:33:24 2017 +0530

    cluster/dht: EBADF handling for fremovexattr and fsetxattr
    
    Add EBADF handling for dht_fremovexattr and dht_fsetxattr.
    
    Change-Id: Ide0d5812dae79655d2565157e5baabcd753b4309
    BUG: 1476665
    Signed-off-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-on: https://review.gluster.org/17999
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Comment 13 Shyamsundar 2017-12-08 12:36:32 EST
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.13.0, please open a new bug report.

glusterfs-3.13.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-December/000087.html
[2] https://www.gluster.org/pipermail/gluster-users/

Note You need to log in before you can comment on or make changes to this bug.