Description of problem: ** time difference between 2 subsequent statedump of glusterfs(fuse mount) process is 2 hours ** statedump2: [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage] size=4294967290 num_allocs=2 max_size=4294967295 max_num_allocs=5 total_allocs=4007351 statedump 3 [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage] size=4294967187 num_allocs=2 max_size=4294967295 max_num_allocs=5 total_allocs=8996888 statedump 4 [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage] size=4294966952 num_allocs=1 max_size=4294967295 max_num_allocs=5 total_allocs=13664729 The mem_type gf_fuse_mt_iov_base consuming almost 4.29 GB but not exceeding the max size for gf_fuse_mt_iov_base. There was no significant increase in memory consumption by gf_fuse_mt_iov_base since statedump 2 but seems to be the reason for glusterfs process being picked up by OOM killer. It also observed that the client system was showing 3.3GB free memory even though there was no rsync' was going on volume mount point. There may be possibility of high memory intensive processes running on the system as well. In fuse_thread_proc() function: iov_in[0].iov_base = GF_CALLOC (1, msg0_size, gf_fuse_mt_iov_base); if (!iobuf || !iov_in[0].iov_base) { gf_log (this->name, GF_LOG_ERROR, "Out of memory"); if (iobuf) iobuf_unref (iobuf); GF_FREE (iov_in[0].iov_base); sleep (10); continue; } ...... if (finh->opcode == FUSE_WRITE) msg = iov_in[1].iov_base; else { if (res > msg0_size) { void *b = GF_REALLOC (iov_in[0].iov_base, res); if (b) { iov_in[0].iov_base = b; finh = (fuse_in_header_t *) iov_in[0].iov_base; } else { gf_log ("glusterfs-fuse", GF_LOG_ERROR, "Out of memory"); send_fuse_err (this, finh, ENOMEM); goto cont_err; } } It seems like we are leaking memory by over-riding iov_in[0].iov_base without freeing the existing memory-ptr in the variable? Re-alloc may return different ptr than the one passed. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
REVIEW: http://review.gluster.org/15052 (mount/fuse: Fix memory leak after re-alloc) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)