Bug 765274 (GLUSTER-3542)

Summary: [glusterfs-3.1.7qa4]: memory leak in nfs server
Product: [Community] GlusterFS Reporter: Raghavendra Bhat <rabhat>
Component: nfsAssignee: Krishna Srinivas <krishna>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: pre-releaseCC: gluster-bugs, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Raghavendra Bhat 2011-09-13 05:55:49 EDT
There is a huge memory leak in nfs server. Just mounting the nfs client, running posix compliance test and dbench for 10 minutes with 100 clients will make valgrind report many leaks in nfs server.

==6776== 1,197,145 bytes in 14,583 blocks are definitely lost in loss record 215 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C3C837: inode_path (inode.c:1003)
==6776==    by 0x7BDC800: nfs_parent_inode_loc_fill (nfs-common.c:320)
==6776==    by 0x7BDC8FC: nfs_entry_loc_fill (nfs-common.c:370)
==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
==6776==    by 0x7BEEF33: nfs3_lookup (nfs3.c:1313)
==6776==    by 0x7BEF0CD: nfs3svc_lookup (nfs3.c:1357)

738,240 bytes in 7,690 blocks are definitely lost in loss record 211 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x7C017DC: __nfs3_get_inode_queue (nfs3-helpers.c:2109)
==6776==    by 0x7C0190D: nfs3_get_inode_queue (nfs3-helpers.c:2132)
==6776==    by 0x7C01A73: nfs3_queue_call_state (nfs3-helpers.c:2183)
==6776==    by 0x7C01B23: __nfs3_file_open_and_resume (nfs3-helpers.c:2205)
==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)

535,644 (104 direct, 535,540 indirect) bytes in 1 blocks are definitely lost in loss record 207 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C526F3: mem_pool_new_fn (mem-pool.c:330)
==6776==    by 0x7C098F2: nfs_rpcsvc_conn_init (rpcsvc.c:697)
==6776==    by 0x7C09DA4: nfs_rpcsvc_conn_listen_init (rpcsvc.c:874)
==6776==    by 0x7C0E266: nfs_rpcsvc_stage_program_register (rpcsvc.c:2799)
==6776==    by 0x7C0E478: nfs_rpcsvc_program_register (rpcsvc.c:2840)
==6776==    by 0x7BDA180: nfs_init_versions (nfs.c:126)
==6776==    by 0x7BDBD4D: init (nfs.c:715)
==6776==    by 0x4C2565C: __xlator_init (xlator.c:809)
==6776==    by 0x4C2577C: xlator_init (xlator.c:837)
==6776==    by 0x4C5CCB8: glusterfs_graph_init (graph.c:328)

77,798 bytes in 1,467 blocks are definitely lost in loss record 197 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x7C05141: gf_strdup (mem-pool.h:91)
==6776==    by 0x7C050B0: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3239)
==6776==    by 0x7BF6F0A: nfs3_rename (nfs3.c:3715)
==6776==    by 0x7BF70C8: nfs3svc_rename (nfs3.c:3755)
==6776==    by 0x7C0C8F4: nfs_rpcsvc_handle_rpc_call (rpcsvc.c:2018)
==6776==    by 0x7C0D87A: nfs_rpcsvc_record_update_state (rpcsvc.c:2507)
==6776==    by 0x7C0D9F0: nfs_rpcsvc_conn_data_poll_in (rpcsvc.c:2550)
==6776==    by 0x7C0DEA1: nfs_rpcsvc_conn_data_handler (rpcsvc.c:2690)
==6776==    by 0x4C517AB: event_dispatch_epoll_handler (event.c:812)
==6776==    by 0x4C5199A: event_dispatch_epoll (event.c:876)

 3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in loss record 176 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C4E6FC: fd_create (fd.c:519)
==6776==    by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228)
==6776==    by 0x7BE6912: nfs_open (nfs-generics.c:177)
==6776==    by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218)
==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)

1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss record 161 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C3BB18: __inode_create (inode.c:513)
==6776==    by 0x4C3BC1F: inode_new (inode.c:544)
==6776==    by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366)
==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
==6776==    by 0x7BF2B32: nfs3_create (nfs3.c:2536)
==6776==    by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573)

==6776== LEAK SUMMARY:
==6776==    definitely lost: 3,510,624 bytes in 35,769 blocks
==6776==    indirectly lost: 582,508 bytes in 24 blocks
==6776==      possibly lost: 38,200,683 bytes in 546 blocks
==6776==    still reachable: 34,368 bytes in 59 blocks
==6776==         suppressed: 0 bytes in 0 blocks
==6776== Reachable blocks (those to which a pointer was found) are not shown.
==6776== To see them, rerun with: --leak-check=full --show-reachable=yes
Comment 1 Krishna Srinivas 2011-09-28 04:58:37 EDT
Investigating memleaks for:

 3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in
loss record 176 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C4E6FC: fd_create (fd.c:519)
==6776==    by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228)
==6776==    by 0x7BE6912: nfs_open (nfs-generics.c:177)
==6776==    by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218)
==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)

1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss
record 161 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C3BB18: __inode_create (inode.c:513)
==6776==    by 0x4C3BC1F: inode_new (inode.c:544)
==6776==    by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366)
==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
==6776==    by 0x7BF2B32: nfs3_create (nfs3.c:2536)
==6776==    by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573)


Leaks for other backtraces are obvious.
Comment 2 Krishna Srinivas 2011-09-29 05:05:08 EDT
The following leaks were not found in valgrind tests for dbench run.

(In reply to comment #1)
> Investigating memleaks for:
> 
>  3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in
> loss record 176 of 221
> ==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
> ==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
> ==6776==    by 0x4C4E6FC: fd_create (fd.c:519)
> ==6776==    by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228)
> ==6776==    by 0x7BE6912: nfs_open (nfs-generics.c:177)
> ==6776==    by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218)
> ==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
> ==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
> ==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
> ==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
> ==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
> ==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
> 
> 1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss
> record 161 of 221
> ==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
> ==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
> ==6776==    by 0x4C3BB18: __inode_create (inode.c:513)
> ==6776==    by 0x4C3BC1F: inode_new (inode.c:544)
> ==6776==    by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366)
> ==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
> ==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
> ==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
> ==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
> ==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
> ==6776==    by 0x7BF2B32: nfs3_create (nfs3.c:2536)
> ==6776==    by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573)
> 
> 
> Leaks for other backtraces are obvious.
Comment 3 Anand Avati 2011-11-15 02:39:33 EST
CHANGE: http://review.gluster.com/533 (Change-Id: I522d4bad71d1e914fa88367f70f5e8c28d64fbbe) merged in master by Vijay Bellur (vijay@gluster.com)