Bug 823616 - Intermittent poorer read performance on 2-way replica mirror.
Intermittent poorer read performance on 2-way replica mirror.
Product: GlusterFS
Classification: Community
Component: replicate (Show other bugs)
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Jeff Darcy
Depends On:
  Show dependency treegraph
Reported: 2012-05-21 13:11 EDT by Jeff Byers
Modified: 2013-07-24 13:43 EDT (History)
2 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-07-24 13:43:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Jeff Byers 2012-05-21 13:11:22 EDT
Intermittent poorer read performance on 2-way replica mirror.

In a GlusterFS configuration where CIFS is being
offered via Samba, and GlusterFS is configured with 2-
way replica mirrors for redundancy, we have been seeing
intermittent poorer read performance. The write
performance seems consistent.

Sometimes the read performance is significantly better
than other times. For example, with 32KB sequential
reads, and 32 KB sequential writes, the poor results I
was seeing were:

  Read:  5.5 MB/s, 175 IOPS, Avg Rsp 183 ms, Max Rsp 227 ms
  Write: 21.0 MB/s, 704 IOPS, Avg Rsp 46 ms, Max Rsp 104 ms


  Read:  29.5 MB/s, 903 IOPS, Avg Rsp 34 ms, Max Rsp 238 ms
  Write: 22.0 MB/s, 725 IOPS, Avg Rsp 44 ms, Max Rsp 106 ms

This has been isolated to be due to the GlusterFS
client on the system where the Samba session arrives
choosing the remote GlusterFS brick to read from
instead of the local one. The replica is in fully sync,
so either node would do, but when the remote node is
selected, there is an additional network hop, etc.

GlusterFS actually seems to favor the remote node, as
if the slower responder wins.

Actions such as triggering a Gluster log rotation seems
to change the current read node.

I tried playing with the GlusterFS option 'read-
subvolume', but this option looks to have been removed in
version 3.1.

GlusterFS should, when possible, favor the local
storage instead of taking an unnecessary network hop.
Comment 1 Jeff Darcy 2012-05-21 13:36:49 EDT
Maybe this would be a good reason to get http://review.gluster.com/#change,3005 moving.
Comment 2 Jeff Darcy 2013-01-21 12:38:34 EST
Fixed by http://review.gluster.org/#change,3005 (merged June 5) and further enhanced by http://review.gluster.org/#change,4234 (merged December 4) which re-enables NUFA to handle the same case for writes.

Note You need to log in before you can comment on or make changes to this bug.