Hide Forgot
There is a huge memory leak in nfs server. Just mounting the nfs client, running posix compliance test and dbench for 10 minutes with 100 clients will make valgrind report many leaks in nfs server. ==6776== 1,197,145 bytes in 14,583 blocks are definitely lost in loss record 215 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x4C3C837: inode_path (inode.c:1003) ==6776== by 0x7BDC800: nfs_parent_inode_loc_fill (nfs-common.c:320) ==6776== by 0x7BDC8FC: nfs_entry_loc_fill (nfs-common.c:370) ==6776== by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075) ==6776== by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132) ==6776== by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150) ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) ==6776== by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244) ==6776== by 0x7BEEF33: nfs3_lookup (nfs3.c:1313) ==6776== by 0x7BEF0CD: nfs3svc_lookup (nfs3.c:1357) 738,240 bytes in 7,690 blocks are definitely lost in loss record 211 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x7C017DC: __nfs3_get_inode_queue (nfs3-helpers.c:2109) ==6776== by 0x7C0190D: nfs3_get_inode_queue (nfs3-helpers.c:2132) ==6776== by 0x7C01A73: nfs3_queue_call_state (nfs3-helpers.c:2183) ==6776== by 0x7C01B23: __nfs3_file_open_and_resume (nfs3-helpers.c:2205) ==6776== by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267) ==6776== by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079) ==6776== by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547) ==6776== by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116) ==6776== by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148) ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) 535,644 (104 direct, 535,540 indirect) bytes in 1 blocks are definitely lost in loss record 207 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x4C526F3: mem_pool_new_fn (mem-pool.c:330) ==6776== by 0x7C098F2: nfs_rpcsvc_conn_init (rpcsvc.c:697) ==6776== by 0x7C09DA4: nfs_rpcsvc_conn_listen_init (rpcsvc.c:874) ==6776== by 0x7C0E266: nfs_rpcsvc_stage_program_register (rpcsvc.c:2799) ==6776== by 0x7C0E478: nfs_rpcsvc_program_register (rpcsvc.c:2840) ==6776== by 0x7BDA180: nfs_init_versions (nfs.c:126) ==6776== by 0x7BDBD4D: init (nfs.c:715) ==6776== by 0x4C2565C: __xlator_init (xlator.c:809) ==6776== by 0x4C2577C: xlator_init (xlator.c:837) ==6776== by 0x4C5CCB8: glusterfs_graph_init (graph.c:328) 77,798 bytes in 1,467 blocks are definitely lost in loss record 197 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x7C05141: gf_strdup (mem-pool.h:91) ==6776== by 0x7C050B0: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3239) ==6776== by 0x7BF6F0A: nfs3_rename (nfs3.c:3715) ==6776== by 0x7BF70C8: nfs3svc_rename (nfs3.c:3755) ==6776== by 0x7C0C8F4: nfs_rpcsvc_handle_rpc_call (rpcsvc.c:2018) ==6776== by 0x7C0D87A: nfs_rpcsvc_record_update_state (rpcsvc.c:2507) ==6776== by 0x7C0D9F0: nfs_rpcsvc_conn_data_poll_in (rpcsvc.c:2550) ==6776== by 0x7C0DEA1: nfs_rpcsvc_conn_data_handler (rpcsvc.c:2690) ==6776== by 0x4C517AB: event_dispatch_epoll_handler (event.c:812) ==6776== by 0x4C5199A: event_dispatch_epoll (event.c:876) 3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in loss record 176 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x4C4E6FC: fd_create (fd.c:519) ==6776== by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228) ==6776== by 0x7BE6912: nfs_open (nfs-generics.c:177) ==6776== by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218) ==6776== by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267) ==6776== by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079) ==6776== by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547) ==6776== by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116) ==6776== by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148) ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) 1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss record 161 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x4C3BB18: __inode_create (inode.c:513) ==6776== by 0x4C3BC1F: inode_new (inode.c:544) ==6776== by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366) ==6776== by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075) ==6776== by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132) ==6776== by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150) ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) ==6776== by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244) ==6776== by 0x7BF2B32: nfs3_create (nfs3.c:2536) ==6776== by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573) ==6776== LEAK SUMMARY: ==6776== definitely lost: 3,510,624 bytes in 35,769 blocks ==6776== indirectly lost: 582,508 bytes in 24 blocks ==6776== possibly lost: 38,200,683 bytes in 546 blocks ==6776== still reachable: 34,368 bytes in 59 blocks ==6776== suppressed: 0 bytes in 0 blocks ==6776== Reachable blocks (those to which a pointer was found) are not shown. ==6776== To see them, rerun with: --leak-check=full --show-reachable=yes
Investigating memleaks for: 3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in loss record 176 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x4C4E6FC: fd_create (fd.c:519) ==6776== by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228) ==6776== by 0x7BE6912: nfs_open (nfs-generics.c:177) ==6776== by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218) ==6776== by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267) ==6776== by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079) ==6776== by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547) ==6776== by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116) ==6776== by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148) ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) 1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss record 161 of 221 ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) ==6776== by 0x4C3BB18: __inode_create (inode.c:513) ==6776== by 0x4C3BC1F: inode_new (inode.c:544) ==6776== by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366) ==6776== by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075) ==6776== by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132) ==6776== by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150) ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) ==6776== by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244) ==6776== by 0x7BF2B32: nfs3_create (nfs3.c:2536) ==6776== by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573) Leaks for other backtraces are obvious.
The following leaks were not found in valgrind tests for dbench run. (In reply to comment #1) > Investigating memleaks for: > > 3,028 (400 direct, 2,628 indirect) bytes in 2 blocks are definitely lost in > loss record 176 of 221 > ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) > ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) > ==6776== by 0x4C4E6FC: fd_create (fd.c:519) > ==6776== by 0x7BE5020: nfs_inode_open (nfs-inodes.c:228) > ==6776== by 0x7BE6912: nfs_open (nfs-generics.c:177) > ==6776== by 0x7C01C76: __nfs3_file_open_and_resume (nfs3-helpers.c:2218) > ==6776== by 0x7C01E9E: nfs3_file_open_and_resume (nfs3-helpers.c:2267) > ==6776== by 0x7BF116F: nfs3_write_open_resume (nfs3.c:2079) > ==6776== by 0x7C02FD4: nfs3_fh_resolve_inode_done (nfs3-helpers.c:2547) > ==6776== by 0x7C04C95: nfs3_fh_resolve_inode (nfs3-helpers.c:3116) > ==6776== by 0x7C04D32: nfs3_fh_resolve_resume (nfs3-helpers.c:3148) > ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) > > 1,632 (1,536 direct, 96 indirect) bytes in 6 blocks are definitely lost in loss > record 161 of 221 > ==6776== at 0x4A05140: calloc (vg_replace_malloc.c:418) > ==6776== by 0x4C52012: __gf_calloc (mem-pool.c:142) > ==6776== by 0x4C3BB18: __inode_create (inode.c:513) > ==6776== by 0x4C3BC1F: inode_new (inode.c:544) > ==6776== by 0x7BDC8E3: nfs_entry_loc_fill (nfs-common.c:366) > ==6776== by 0x7C04A25: nfs3_fh_resolve_entry_hard (nfs3-helpers.c:3075) > ==6776== by 0x7C04CDE: nfs3_fh_resolve_entry (nfs3-helpers.c:3132) > ==6776== by 0x7C04D40: nfs3_fh_resolve_resume (nfs3-helpers.c:3150) > ==6776== by 0x7C04F2C: nfs3_fh_resolve_root (nfs3-helpers.c:3202) > ==6776== by 0x7C050D7: nfs3_fh_resolve_and_resume (nfs3-helpers.c:3244) > ==6776== by 0x7BF2B32: nfs3_create (nfs3.c:2536) > ==6776== by 0x7BF2CEA: nfs3svc_create (nfs3.c:2573) > > > Leaks for other backtraces are obvious.
CHANGE: http://review.gluster.com/533 (Change-Id: I522d4bad71d1e914fa88367f70f5e8c28d64fbbe) merged in master by Vijay Bellur (vijay)