Description of problem: Across GlusterFS code there could be few memory leaks and reference leaks which may end up consuming lots of system memory/resources and are very tough to debug. There are few tools like valgrind etc available which report leaks from the time of process start till it exits. But that's not sufficient at times. We also need to be able to debug and find out these leaks in the live system if needed. So we have decided to use systemtap scripts (already widely used for kernel modules) to achieve the same. This bug is to track the various approaches being discussed and provide the scripts to be used for debugging. Version-Release number of selected component (if applicable): Mainline How reproducible: N/A Steps to Reproduce: 1. Run stap script which probes memory allocators and deallocators like dict_ref() and dict_unref(). 2. Perform io's on the corresponding probe process. 3. Analyze the output to check if there are any leaks Actual results: There should not be any leaks reported after the I/Os are finished. Expected results: Traces of the function from where leakage occurs. Additional info:
Hi Sonal, There are some systemtap scripts floating around to debug different parts in Gluster. I think it is a good idea to collect and improve them. Where do you want to store the scripts that will be written? Also, are you planning to write scripts, or is this just a bug to share the idea? Thanks, Niels See-also: https://lists.gnu.org/archive/html/gluster-devel/2014-03/msg00161.html
(In reply to Niels de Vos from comment #1) > Hi Sonal, > > There are some systemtap scripts floating around to debug different parts in > Gluster. I think it is a good idea to collect and improve them. Where do you > want to store the scripts that will be written? Also, are you planning to > write scripts, or is this just a bug to share the idea? > > Thanks, > Niels > > > See-also: > https://lists.gnu.org/archive/html/gluster-devel/2014-03/msg00161.html Hey Neils, I saw the referred script. Its a good idea to work on them as i am playing with stap these days. But currently i am focusing on the above issue(Reference Leakage). May be after solving this i can try my hands on them. I am writing stap script to see the possible leaks and to resolve them further. Storing the scripts,as in? Thanks Best -Sonal
This bug gave me the impression that you wanted to add (systemtap) scripts that are used for debugging and improving Gluster, and not a bug for the actual fixes that are implemented after the scripts have been used. I think there is a lot of value in gathering these script and have them as part of the glusterfs sources, or maybe as a separate glusterfs-debug-utils project on GitHub. So, where do you want to collect these scripts so that others can find them and apply the ideas for tracking down similar problems?
(In reply to Niels de Vos from comment #3) > This bug gave me the impression that you wanted to add (systemtap) scripts > that are used for debugging and improving Gluster, and not a bug for the > actual fixes that are implemented after the scripts have been used. > > I think there is a lot of value in gathering these script and have them as > part of the glusterfs sources, or maybe as a separate glusterfs-debug-utils > project on GitHub. So, where do you want to collect these scripts so that > others can find them and apply the ideas for tracking down similar problems? Ya, you got it right. The script is to improve gluster performance. I am working on the scripts. I will be putting it on GitHub probably, as a separate project, so that anyone can utilize it for gluster. Thanks Regards -Sonal
When you have something for sharing, please request a repository nuder https://github.com/gluster/ by reporting a bug against the project-infrastructure. The infra team can then create one for you and give permissions to push changes there.
(In reply to Niels de Vos from comment #5) > When you have something for sharing, please request a repository nuder > https://github.com/gluster/ by reporting a bug against the > project-infrastructure. The infra team can then create one for you and give > permissions to push changes there. sure :)
REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#1) for review on master by Sonal Arora (sarora)
REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#2) for review on master by Sonal Arora (sarora)
REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#3) for review on master by Sonal Arora (sarora)
REVIEW: https://review.gluster.org/17368 (../extras/ref-leak-identify.stp : To identify reference leak paths in glusterfs.) posted (#4) for review on master by Sonal Arora (sarora)
We have moved to automated ASan jobs which point at leaks. https://build.gluster.org/job/asan/ Not pursuing it further!