Bug 809034

Summary:	mem-pool logs filling up client log file
Product:	[Community] GlusterFS	Reporter:	Anush Shetty <ashetty>
Component:	logging	Assignee:	Amar Tumballi <amarts>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Ujjwala <ujjwala>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	mainline	CC:	gluster-bugs, sdharane, vraman
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-07-24 17:44:44 UTC	Type:	---
Regression:	---	Mount Type:	fuse
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	817967

Description Anush Shetty 2012-04-02 10:10:11 UTC

Description of problem: The fuse client log file in DEBUG mode is filled with log messages when mem pool is full


Version-Release number of selected component (if applicable): upstream

  
Actual results:

[2012-04-02 15:31:38.924422] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste
r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c
2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924478] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste
r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c
2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924533] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste
r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924588] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924644] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924699] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924755] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.925128] D [client3_1-fops.c:2758:client_fdctx_destroy] 0-test2-client-0: sending releasedir on fd
[2012-04-02 15:31:38.928907] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.928999] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.929057] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem


Expected results: Shouldn't be filling up so much. 


Additional info:

Comment 1 Amar Tumballi 2012-04-02 10:29:22 UTC

Anush, Is it possible to take a 'statedump' (with kill -USR1) of the process when this happens? I want the numbers of mempool structure variables to set to right values.

Comment 2 Anush Shetty 2012-04-02 11:28:15 UTC

pool-name=glusterfs:dict_t
hot-count=29
cold-count=995
padded_sizeof=84
alloc-count=314959
max-alloc=1024
pool-misses=82258
max-stdalloc=2548

Comment 3 Amar Tumballi 2012-04-11 18:14:48 UTC

looks like if we increase the dict_t pool count to 4096, we would not hit these logs so frequently. will send out a patch for this.

Comment 4 Anand Avati 2012-04-11 23:48:32 UTC

CHANGE: http://review.gluster.com/3131 (glusterfsd: mem-pool count of dict_t is adjusted) merged in master by Anand Avati (avati)

Comment 5 Anush Shetty 2012-05-23 12:08:07 UTC

Verified with 3.3.0qa42