Bug 1250170

Summary:	Write performance from a Windows client on 3-way replicated volume decreases substantially when one brick in the replica set is brought down
Product:	[Community] GlusterFS	Reporter:	Ravishankar N <ravishankar>
Component:	replicate	Assignee:	Ravishankar N <ravishankar>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bengland, jharriga, mpillai
Target Milestone:	---	Keywords:	Reopened, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.8rc2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1227759
Clones:	1255698 (view as bug list)		Environment:
Last Closed:	2016-06-16 13:28:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1227759
Bug Blocks:	1216951, 1223636, 1255698

Description Ravishankar N 2015-08-04 16:05:51 UTC

+++ This bug was initially created as a clone of Bug #1227759 +++

Description of problem:
-------------------------

On a 3-way replicated volume mounted on a Windows client, one of the bricks was brought down. The write speed observed while copying a .iso file was substantially reduced as compared to the speeds observed with all bricks in the replica set being up.

See results of the copy operations below -

With all 3 bricks up
--------------------

PS Z:\> robocopy C:\Users\shruti\Downloads\test Z:\ RHS.iso

------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
------------------------------------------------------------------------

  Started : Wed Jun 03 16:36:45 2015

   Source : C:\Users\shruti\Downloads\test\
     Dest : Z:\

    Files : RHS.iso

  Options : /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------

                           1    C:\Users\shruti\Downloads\test\
100%        New File               1.5 g        RHS.iso

------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         1         0         0         0
   Files :         1         1         0         0         0         0
   Bytes :   1.566 g   1.566 g         0         0         0         0
   Times :   0:01:21   0:01:21                       0:00:00   0:00:00


   Speed :            20750024 Bytes/sec.
   Speed :            1187.325 MegaBytes/min.

   Ended : Wed Jun 03 16:38:06 2015


With one brick in replica set brought down
-------------------------------------------

PS Z:\> robocopy C:\Users\shruti\Downloads\test Z:\ RHS.iso

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Wed Jun 03 16:39:38 2015

   Source : C:\Users\shruti\Downloads\test\
     Dest : Z:\

    Files : RHS.iso

  Options : /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------------

                           1    C:\Users\shruti\Downloads\test\
100%        New File               1.5 g        RHS.iso

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         1         0         0         0
   Files :         1         1         0         0         0         0
   Bytes :   1.566 g   1.566 g         0         0         0         0
   Times :   0:03:26   0:03:26                       0:00:00   0:00:00


   Speed :             8158524 Bytes/sec.
   Speed :             466.834 MegaBytes/min.

   Ended : Wed Jun 03 16:43:05 2015

Such a drop in performance is not observed with a 2-way replicated volume in the same setup.

Version-Release number of selected component (if applicable):
----------------------------------------------------------------
glusterfs-3.7.0-3.el6rhs.x86_64
samba-4.1.17-7.el6rhs.x86_64

How reproducible:
------------------
100%

Steps to Reproduce:
--------------------
1. Created a 6x3 volume and mounted on a Windows 7 client machine.
2. Copied a .iso file from the Windows local drive to the gluster share.
3. Brought down one brick of the replica set and ran the copy operation again.

Actual results:
-----------------

There is a significant drop in write speeds when results from step 3 are compared to results from step 2. Write speed with one brick down reduce to roughly 40% of the speed observed with all bricks up.

Expected results:
------------------

Expected behaviour is that write speed should not be affected if one brick in a replica set is down.

Comment 1 Ravishankar N 2015-08-04 16:09:36 UTC

Patch: http://review.gluster.org/#/c/11827/

Comment 2 Anand Avati 2015-08-05 17:00:35 UTC

REVIEW: http://review.gluster.org/11827 (afr: modify afr_txn_nothing_failed()) posted (#2) for review on master by Ravishankar N (ravishankar)

Comment 3 Ben England 2015-08-10 01:33:50 UTC

adding perf team members to cc list

Comment 4 Nagaprasad Sathyanarayana 2015-10-25 14:58:26 UTC

Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 5 Niels de Vos 2016-06-16 13:28:07 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user