Bug 1362010

Summary:	Memory leak observed with upcall polling
Product:	[Community] GlusterFS	Reporter:	Soumya Koduri <skoduri>
Component:	upcall	Assignee:	Soumya Koduri <skoduri>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.7.13	CC:	bugs, ndevos
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	glusterfs-3.7.15	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1358608	Environment:
Last Closed:	2016-09-01 09:20:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1358608
Bug Blocks:	1338068, 1361665

Description Soumya Koduri 2016-08-01 07:48:12 UTC

+++ This bug was initially created as a clone of Bug #1358608 +++

Description of problem:

There is memory leak observed when there are large number of upcall notifications generated. Ideally after polling them, all the upcall events generated should be cleaned up. But that wasn't the case. There were few upcall events not cleaned up and resulted in memory leak.

The issue is that during polling (currently), upcall entries are not removed under mutex lock. This resulted in the list corruption leaving few entries from being removed.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Vijay Bellur on 2016-07-21 02:54:30 EDT ---

REVIEW: http://review.gluster.org/14972 (glfs/upcall: entries should be removed under mutex lock) posted (#1) for review on master by soumya k (skoduri)

--- Additional comment from Vijay Bellur on 2016-07-21 03:06:22 EDT ---

REVIEW: http://review.gluster.org/14972 (glfs/upcall: entries should be removed under mutex lock) posted (#2) for review on master by soumya k (skoduri)

--- Additional comment from Vijay Bellur on 2016-07-22 05:08:58 EDT ---

REVIEW: http://review.gluster.org/14984 (gfapi/upcall: Fix a ref leak) posted (#1) for review on master by soumya k (skoduri)

--- Additional comment from Vijay Bellur on 2016-07-22 11:13:33 EDT ---

COMMIT: http://review.gluster.org/14972 committed in master by Jeff Darcy (jdarcy) 
------
commit 89dee8b46e126bc1d7541da90fa60844aa83451e
Author: Soumya Koduri <skoduri>
Date:   Thu Jul 21 12:14:27 2016 +0530

    glfs/upcall: entries should be removed under mutex lock
    
    During poll, upcall entries should be removed from the
    upcall_list only under upcall_list_mutex lock. Otherwise
    it could result in the list corruption if there are entries
    being added during poll resulting in memory leak.
    
    Also addressed a probable leak during any failures with upcall
    entry addition.
    
    Change-Id: I468183f961eb6faed9a0a1bcb783705f711641fc
    BUG: 1358608
    Signed-off-by: Soumya Koduri <skoduri>
    Reviewed-on: http://review.gluster.org/14972
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jdarcy>

--- Additional comment from Vijay Bellur on 2016-07-24 05:55:32 EDT ---

COMMIT: http://review.gluster.org/14984 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit bb48eb46910085928efbd7fb491c5b2db25bba98
Author: Soumya Koduri <skoduri>
Date:   Fri Jul 22 13:15:58 2016 +0530

    gfapi/upcall: Fix a ref leak
    
    inode_find (used to create the handle) takes a reference
    of the inode. This needs to be un'refernced to avoid leak.
    
    Change-Id: I22f03577a8f1d9608cfc62d57202cfc4c2ba12b3
    BUG: 1358608
    Signed-off-by: Soumya Koduri <skoduri>
    Reviewed-on: http://review.gluster.org/14984
    Reviewed-by: jiffin tony Thottan <jthottan>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jdarcy>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.org>

Comment 1 Vijay Bellur 2016-08-01 08:51:57 UTC

REVIEW: http://review.gluster.org/15057 (gfapi/upcall: Fix a ref leak) posted (#1) for review on release-3.7 by soumya k (skoduri)

Comment 2 Vijay Bellur 2016-08-01 08:53:04 UTC

REVIEW: http://review.gluster.org/15058 (glfs/upcall: entries should be removed under mutex lock) posted (#1) for review on release-3.7 by soumya k (skoduri)

Comment 3 Vijay Bellur 2016-08-02 09:18:58 UTC

COMMIT: http://review.gluster.org/15058 committed in release-3.7 by Niels de Vos (ndevos) 
------
commit 3fdf80501f9539067a0801ad279446a4be9013d3
Author: Soumya Koduri <skoduri>
Date:   Thu Jul 21 12:14:27 2016 +0530

    glfs/upcall: entries should be removed under mutex lock
    
    During poll, upcall entries should be removed from the
    upcall_list only under upcall_list_mutex lock. Otherwise
    it could result in the list corruption if there are entries
    being added during poll resulting in memory leak.
    
    Also addressed a probable leak during any failures with upcall
    entry addition.
    
    This is backport of below master patch -
       http://review.gluster.org/14972
    
    >Change-Id: I468183f961eb6faed9a0a1bcb783705f711641fc
    >BUG: 1358608
    >Signed-off-by: Soumya Koduri <skoduri>
    >Reviewed-on: http://review.gluster.org/14972
    >Reviewed-by: Shyamsundar Ranganathan <srangana>
    >Reviewed-by: Jeff Darcy <jdarcy>
    >(cherry picked from commit 89dee8b46e126bc1d7541da90fa60844aa83451e)
    
    Change-Id: Ib6702911ca1fa09a3f1a493b27195665603854d3
    BUG: 1362010
    Signed-off-by: Soumya Koduri <skoduri>
    Reviewed-on: http://review.gluster.org/15058
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Tested-by: Oleksandr Natalenko <oleksandr>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Niels de Vos <ndevos>

Comment 4 Vijay Bellur 2016-08-03 09:31:15 UTC

COMMIT: http://review.gluster.org/15057 committed in release-3.7 by Niels de Vos (ndevos) 
------
commit 1f97cc171aa97c4e6941d2c46ed6285d371fda10
Author: Soumya Koduri <skoduri>
Date:   Fri Jul 22 13:15:58 2016 +0530

    gfapi/upcall: Fix a ref leak
    
    inode_find (used to create the handle) takes a reference
    of the inode. This needs to be un'refernced to avoid leak.
    
    This is backport of below master patch -
       http://review.gluster.org/14984
    
    >Change-Id: I22f03577a8f1d9608cfc62d57202cfc4c2ba12b3
    >BUG: 1358608
    >Signed-off-by: Soumya Koduri <skoduri>
    >Reviewed-on: http://review.gluster.org/14984
    >Reviewed-by: jiffin tony Thottan <jthottan>
    >Reviewed-by: Jeff Darcy <jdarcy>
    >(cherry picked from commit bb48eb46910085928efbd7fb491c5b2db25bba98)
    
    Change-Id: Iaa91ee757e3497e25d8669c1592106b6266057a0
    BUG: 1362010
    Signed-off-by: Soumya Koduri <skoduri>
    Reviewed-on: http://review.gluster.org/15057
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Tested-by: Oleksandr Natalenko <oleksandr>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    Reviewed-by: Niels de Vos <ndevos>

Comment 5 Kaushal 2016-09-01 09:20:59 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.15, please open a new bug report.

glusterfs-3.7.15 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-devel/2016-September/050714.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 6 Kaushal 2016-09-01 09:33:03 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.15, please open a new bug report.

glusterfs-3.7.15 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-devel/2016-September/050714.html
[2] https://www.gluster.org/pipermail/gluster-users/