Bug 765274 - (GLUSTER-3542) [glusterfs-3.1.7qa4]: memory leak in nfs server
[glusterfs-3.1.7qa4]: memory leak in nfs server
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
pre-release
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Krishna Srinivas
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-09-13 05:55 EDT by Raghavendra Bhat
Modified: 2015-12-01 11:45 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Raghavendra Bhat 2011-09-13 05:55:49 EDT
There is a huge memory leak in nfs server. Just mounting the nfs client, running posix compliance test and dbench for 10 minutes with 100 clients will make valgrind report many leaks in nfs server.

==6776== 1,197,145 bytes in 14,583 blocks are definitely lost in loss record 215 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C3C837: inode_path (inode.c:1003)
==6776==    by 0x7BDC800: nfs_parent_inode_loc_fill (nfs-common.c:320)
==6776==    by 0x7BDC8FC: nfs_entry_loc_fill (nfs-common.c:370)
==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
==6776==    by 0x7BEEF33: nfs3_lookup (nfs3.c:1313)
==6776==    by 0x7BEF0CD: nfs3svc_lookup (nfs3.c:1357)

738,240 bytes in 7,690 blocks are definitely lost in loss record 211 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x7C017DC: __nfs3_get_inode_queue (nfs3-helpers.c:2109)
==6776==    by 0x7C0190D: nfs3_get_inode_queue (nfs3-helpers.c:2132)
==6776==    by 0x7C01A73: nfs3_queue_call_state (nfs3-helpers.c:2183)
==6776==    by 0x7C01B23: __nfs3_file_open_and_resume (nfs3-helpers.c:2205)
==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)

535,644 (104 direct, 535,540 indirect) bytes in 1 blocks are definitely lost in loss record 207 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C526F3: mem_pool_new_fn (mem-pool.c:330)
==6776==    by 0x7C098F2: nfs_rpcsvc_conn_init (rpcsvc.c:697)
==6776==    by 0x7C09DA4: nfs_rpcsvc_conn_listen_init (rpcsvc.c:874)
==6776==    by 0x7C0E266: nfs_rpcsvc_stage_program_register (rpcsvc.c:2799)
==6776==    by 0x7C0E478: nfs_rpcsvc_program_register (rpcsvc.c:2840)
==6776==    by 0x7BDA180: nfs_init_versions (nfs.c:126)
==6776==    by 0x7BDBD4D: init (nfs.c:715)
==6776==    by 0x4C2565C: __xlator_init (xlator.c:809)
==6776==    by 0x4C2577C: xlator_init (xlator.c:837)
==6776==    by 0x4C5CCB8: glusterfs_graph_init (graph.c:328)

77,798 bytes in 1,467 blocks are definitely lost in loss record 197 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x7C05141: gf_strdup (mem-pool.h:91)
==6776==    by 0x7C050B0: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3239)
==6776==    by 0x7BF6F0A: nfs3_rename (nfs3.c:3715)
==6776==    by 0x7BF70C8: nfs3svc_rename (nfs3.c:3755)
==6776==    by 0x7C0C8F4: nfs_rpcsvc_handle_rpc_call (rpcsvc.c:2018)
==6776==    by 0x7C0D87A: nfs_rpcsvc_record_update_state (rpcsvc.c:2507)
==6776==    by 0x7C0D9F0: nfs_rpcsvc_conn_data_poll_in (rpcsvc.c:2550)
==6776==    by 0x7C0DEA1: nfs_rpcsvc_conn_data_handler (rpcsvc.c:2690)
==6776==    by 0x4C517AB: event_dispatch_epoll_handler (event.c:812)
==6776==    by 0x4C5199A: event_dispatch_epoll (event.c:876)

 3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in loss record 176 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C4E6FC: fd_create (fd.c:519)
==6776==    by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228)
==6776==    by 0x7BE6912: nfs_open (nfs-generics.c:177)
==6776==    by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218)
==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)

1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss record 161 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C3BB18: __inode_create (inode.c:513)
==6776==    by 0x4C3BC1F: inode_new (inode.c:544)
==6776==    by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366)
==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
==6776==    by 0x7BF2B32: nfs3_create (nfs3.c:2536)
==6776==    by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573)

==6776== LEAK SUMMARY:
==6776==    definitely lost: 3,510,624 bytes in 35,769 blocks
==6776==    indirectly lost: 582,508 bytes in 24 blocks
==6776==      possibly lost: 38,200,683 bytes in 546 blocks
==6776==    still reachable: 34,368 bytes in 59 blocks
==6776==         suppressed: 0 bytes in 0 blocks
==6776== Reachable blocks (those to which a pointer was found) are not shown.
==6776== To see them, rerun with: --leak-check=full --show-reachable=yes
Comment 1 Krishna Srinivas 2011-09-28 04:58:37 EDT
Investigating memleaks for:

 3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in
loss record 176 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C4E6FC: fd_create (fd.c:519)
==6776==    by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228)
==6776==    by 0x7BE6912: nfs_open (nfs-generics.c:177)
==6776==    by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218)
==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)

1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss
record 161 of 221
==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
==6776==    by 0x4C3BB18: __inode_create (inode.c:513)
==6776==    by 0x4C3BC1F: inode_new (inode.c:544)
==6776==    by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366)
==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
==6776==    by 0x7BF2B32: nfs3_create (nfs3.c:2536)
==6776==    by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573)


Leaks for other backtraces are obvious.
Comment 2 Krishna Srinivas 2011-09-29 05:05:08 EDT
The following leaks were not found in valgrind tests for dbench run.

(In reply to comment #1)
> Investigating memleaks for:
> 
>  3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in
> loss record 176 of 221
> ==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
> ==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
> ==6776==    by 0x4C4E6FC: fd_create (fd.c:519)
> ==6776==    by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228)
> ==6776==    by 0x7BE6912: nfs_open (nfs-generics.c:177)
> ==6776==    by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218)
> ==6776==    by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267)
> ==6776==    by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079)
> ==6776==    by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547)
> ==6776==    by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116)
> ==6776==    by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148)
> ==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
> 
> 1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss
> record 161 of 221
> ==6776==    at 0x4A05140: calloc (vg_replace_malloc.c:418)
> ==6776==    by 0x4C52012: __gf_calloc (mem-pool.c:142)
> ==6776==    by 0x4C3BB18: __inode_create (inode.c:513)
> ==6776==    by 0x4C3BC1F: inode_new (inode.c:544)
> ==6776==    by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366)
> ==6776==    by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075)
> ==6776==    by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132)
> ==6776==    by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150)
> ==6776==    by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202)
> ==6776==    by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244)
> ==6776==    by 0x7BF2B32: nfs3_create (nfs3.c:2536)
> ==6776==    by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573)
> 
> 
> Leaks for other backtraces are obvious.
Comment 3 Anand Avati 2011-11-15 02:39:33 EST
CHANGE: http://review.gluster.com/533 (Change-Id: I522d4bad71d1e914fa88367f70f5e8c28d64fbbe) merged in master by Vijay Bellur (vijay@gluster.com)

Note You need to log in before you can comment on or make changes to this bug.