Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1238476

Summary:	Throttle background heals in disperse volumes
Product:	[Community] GlusterFS	Reporter:	Pranith Kumar K <pkarampu>
Component:	disperse	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.7.2	CC:	bugs, gluster-bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.3	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1237381	Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1237381
Bug Blocks:

Description Pranith Kumar K 2015-07-02 00:13:21 UTC

+++ This bug was initially created as a clone of Bug #1237381 +++

Description of problem:
If the number of background heals increase then the mount can die because of OOM killers. Number of background heals need to be throttled.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Anand Avati on 2015-06-30 22:09:29 EDT ---

REVIEW: http://review.gluster.org/11471 (cluster/ec: Add throttling in background healing) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-06-30 22:09:44 EDT ---

REVIEW: http://review.gluster.org/11472 (cluster/ec: Remove dead code) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-06-30 22:09:55 EDT ---

REVIEW: http://review.gluster.org/11473 (cluster/ec: Make background healing optional behavior) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-07-01 13:42:49 EDT ---

REVIEW: http://review.gluster.org/11495 (libglusterfs: Use GF_CALLOC/GF_FREE instead of CALLOC/FREE) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-07-01 15:18:53 EDT ---

COMMIT: http://review.gluster.org/11472 committed in master by Vijay Bellur (vbellur) 
------
commit c66026b9bf521172f49ce36a5a7b94fae1bbf267
Author: Pranith Kumar K <pkarampu>
Date:   Tue Jun 30 22:09:57 2015 +0530

    cluster/ec: Remove dead code
    
    Change-Id: I99d7a038f29cebe823e17a8dda40d335441185bc
    BUG: 1237381
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/11472
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Xavier Hernandez <xhernandez>
    Tested-by: NetBSD Build System <jenkins.org>

--- Additional comment from Anand Avati on 2015-07-01 20:10:02 EDT ---

COMMIT: http://review.gluster.org/11471 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit fb20db2078dd7fe1a202f8c0c6f8bd7ecc7ff875
Author: Pranith Kumar K <pkarampu>
Date:   Tue Jun 30 18:45:36 2015 +0530

    cluster/ec: Add throttling in background healing
    
    - 8 parallel heals can happen.
    - 128 heals will wait for their turn
    - Heals will be rejected if 128 heals are already waiting.
    
    Change-Id: I2e99bf064db7bce71838ed9901a59ffd565ac390
    BUG: 1237381
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/11471
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Xavier Hernandez <xhernandez>
    Tested-by: NetBSD Build System <jenkins.org>

Comment 1 Anand Avati 2015-07-02 00:13:59 UTC

REVIEW: http://review.gluster.org/11497 (cluster/ec: Remove dead code) posted (#1) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu)

Comment 2 Anand Avati 2015-07-02 07:41:23 UTC

REVIEW: http://review.gluster.org/11501 (libglusterfs: Use GF_CALLOC/GF_FREE instead of CALLOC/FREE) posted (#1) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu)

Comment 3 Anand Avati 2015-07-09 05:00:35 UTC

COMMIT: http://review.gluster.org/11501 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit cd059fb2bf00f5fcb72dac2379db34a5622f1d08
Author: Pranith Kumar K <pkarampu>
Date:   Wed Jul 1 22:32:44 2015 +0530

    libglusterfs: Use GF_CALLOC/GF_FREE instead of CALLOC/FREE
    
    - Also removed numbers for the types as the string form of type is printed in
      statedump now, so the numbers are not needed anymore.
    
     >Change-Id: I6e8c15a1dc8cb6187842f96f1d46ec0f26a602b4
     >BUG: 1237381
     >Signed-off-by: Pranith Kumar K <pkarampu>
     >Reviewed-on: http://review.gluster.org/11495
     >Tested-by: Gluster Build System <jenkins.com>
     >Tested-by: NetBSD Build System <jenkins.org>
     >Reviewed-by: Vijay Bellur <vbellur>
     >(cherry picked from commit e7f2547f89dbcd90cdb3714f63620a36bdc2ef3a)
    
    Change-Id: I60dbe14ba0c9855982e235001ee86747e3fa87a0
    BUG: 1238476
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/11501
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>

Comment 4 Anand Avati 2015-07-21 11:54:14 UTC

REVIEW: http://review.gluster.org/11679 (cluster/ec: Add throttling in background healing) posted (#2) for review on release-3.7 by Xavier Hernandez (xhernandez)

Comment 5 Anand Avati 2015-07-21 21:35:40 UTC

COMMIT: http://review.gluster.org/11679 committed in release-3.7 by Xavier Hernandez (xhernandez) 
------
commit 05f7cc9815a9fe067584251fa89b1461938f41ea
Author: Pranith Kumar K <pkarampu>
Date:   Tue Jun 30 18:45:36 2015 +0530

    cluster/ec: Add throttling in background healing
    
    - 8 parallel heals can happen.
    - 128 heals will wait for their turn
    - Heals will be rejected if 128 heals are already waiting.
    
     >Change-Id: I2e99bf064db7bce71838ed9901a59ffd565ac390
     >BUG: 1237381
     >Signed-off-by: Pranith Kumar K <pkarampu>
     >Reviewed-on: http://review.gluster.org/11471
     >Tested-by: Gluster Build System <jenkins.com>
     >Reviewed-by: Xavier Hernandez <xhernandez>
     >Tested-by: NetBSD Build System <jenkins.org>
    
    BUG: 1238476
    Change-Id: Id9625536197c1f5d6c3a9ed53f80bd9e6d14c507
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/11679
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Xavier Hernandez <xhernandez>

Comment 6 Anand Avati 2015-07-21 21:37:41 UTC

COMMIT: http://review.gluster.org/11680 committed in release-3.7 by Xavier Hernandez (xhernandez) 
------
commit 918b3aeae03b3aecc64fbc202f00a7c1955f6db7
Author: Pranith Kumar K <pkarampu>
Date:   Tue Jun 30 23:01:36 2015 +0530

    cluster/ec: Make background healing optional behavior
    
    Provide options to control number of active background heal count and qlen.
    
     >Change-Id: Idc2419219d881f47e7d2e9bbc1dcdd999b372033
     >BUG: 1237381
     >Signed-off-by: Pranith Kumar K <pkarampu>
     >Reviewed-on: http://review.gluster.org/11473
     >Reviewed-by: Xavier Hernandez <xhernandez>
     >Tested-by: Gluster Build System <jenkins.com>
    
    BUG: 1238476
    Change-Id: I22ba902d9911195656db9e458c01b54cf0afcd7a
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/11680
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Xavier Hernandez <xhernandez>

Comment 7 Kaushal 2015-07-30 09:47:41 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.3, please open a new bug report.

glusterfs-3.7.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/12078
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 8 Kaushal 2015-07-30 09:47:43 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.3, please open a new bug report.

glusterfs-3.7.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/12078
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 9 Kaushal 2015-07-30 09:47:51 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.3, please open a new bug report.

glusterfs-3.7.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/12078
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 10 Kaushal 2015-07-30 09:50:10 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.3, please open a new bug report.

glusterfs-3.7.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/12078
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user