Bug 1417539

Summary:	Systemtap scripts to monitor leaks across GlusterFS protocol
Product:	[Community] GlusterFS	Reporter:	Sonal <sarora>
Component:	core	Assignee:	Sonal <sarora>
Status:	CLOSED DEFERRED	QA Contact:
Severity:	high	Docs Contact:
Priority:	low
Version:	mainline	CC:	atumball, bugs, ndevos, pasik, sarora
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-10-24 10:24:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sonal 2017-01-30 07:35:55 UTC

Description of problem:
Across GlusterFS code there could be few memory leaks and reference leaks which may end up consuming lots of system memory/resources and are very tough to debug. There are few tools like valgrind etc available which report leaks from the time of process start till it exits. But that's not sufficient at times. We also need to be able to debug and find out these leaks in the live system if needed.

So we have decided to use systemtap scripts (already widely used for kernel modules) to achieve the same. This bug is to track the various approaches being discussed and provide the scripts to be used for debugging.

Version-Release number of selected component (if applicable):
Mainline

How reproducible:
N/A

Steps to Reproduce:
1. Run stap script which probes memory allocators and deallocators like dict_ref() and dict_unref().
2. Perform io's on the corresponding probe process.
3. Analyze the output to check if there are any leaks

Actual results:
There should not be any leaks reported after the I/Os are finished.

Expected results:
Traces of the function from where leakage occurs.

Additional info:

Comment 1 Niels de Vos 2017-01-30 13:51:48 UTC

Hi Sonal,

There are some systemtap scripts floating around to debug different parts in Gluster. I think it is a good idea to collect and improve them. Where do you want to store the scripts that will be written? Also, are you planning to write scripts, or is this just a bug to share the idea?

Thanks,
Niels


See-also: https://lists.gnu.org/archive/html/gluster-devel/2014-03/msg00161.html

Comment 2 Sonal 2017-01-30 14:33:47 UTC

(In reply to Niels de Vos from comment #1)
> Hi Sonal,
> 
> There are some systemtap scripts floating around to debug different parts in
> Gluster. I think it is a good idea to collect and improve them. Where do you
> want to store the scripts that will be written? Also, are you planning to
> write scripts, or is this just a bug to share the idea?
> 
> Thanks,
> Niels
> 
> 
> See-also:
> https://lists.gnu.org/archive/html/gluster-devel/2014-03/msg00161.html

Hey Neils,

I saw the referred script. Its a good idea to work on them as i am playing with stap these days. But currently i am focusing on the above issue(Reference Leakage). May be after solving this i can try my hands on them.

I am writing stap script to see the possible leaks and to resolve them further.

Storing the scripts,as in?

Thanks

Best
-Sonal

Comment 3 Niels de Vos 2017-01-30 15:30:51 UTC

This bug gave me the impression that you wanted to add (systemtap) scripts that are used for debugging and improving Gluster, and not a bug for the actual fixes that are implemented after the scripts have been used.

I think there is a lot of value in gathering these script and have them as part of the glusterfs sources, or maybe as a separate glusterfs-debug-utils project on GitHub. So, where do you want to collect these scripts so that others can find them and apply the ideas for tracking down similar problems?

Comment 4 Sonal 2017-02-15 06:16:14 UTC

(In reply to Niels de Vos from comment #3)
> This bug gave me the impression that you wanted to add (systemtap) scripts
> that are used for debugging and improving Gluster, and not a bug for the
> actual fixes that are implemented after the scripts have been used.
> 
> I think there is a lot of value in gathering these script and have them as
> part of the glusterfs sources, or maybe as a separate glusterfs-debug-utils
> project on GitHub. So, where do you want to collect these scripts so that
> others can find them and apply the ideas for tracking down similar problems?

Ya, you got it right. The script is to improve gluster performance. 
I am working on the scripts. I will be putting it on GitHub probably, as a separate project, so that anyone can utilize it for gluster.

Thanks

Regards
-Sonal

Comment 5 Niels de Vos 2017-02-15 09:23:49 UTC

When you have something for sharing, please request a repository nuder https://github.com/gluster/ by reporting a bug against the project-infrastructure. The infra team can then create one for you and give permissions to push changes there.

Comment 6 Sonal 2017-02-20 09:33:21 UTC

(In reply to Niels de Vos from comment #5)
> When you have something for sharing, please request a repository nuder
> https://github.com/gluster/ by reporting a bug against the
> project-infrastructure. The infra team can then create one for you and give
> permissions to push changes there.

sure :)

Comment 7 Worker Ant 2017-05-23 10:22:36 UTC

REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#1) for review on master by Sonal Arora (sarora)

Comment 8 Worker Ant 2017-05-23 10:27:55 UTC

REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#2) for review on master by Sonal Arora (sarora)

Comment 9 Worker Ant 2017-05-23 10:36:58 UTC

REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#3) for review on master by Sonal Arora (sarora)

Comment 10 Worker Ant 2017-05-25 08:09:06 UTC

REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#4) for review on master by Sonal Arora (sarora)

Comment 11 Amar Tumballi 2018-10-24 10:24:30 UTC

We have moved to automated ASan jobs which point at leaks. https://build.gluster.org/job/asan/ Not pursuing it further!