Bug 809034 - mem-pool logs filling up client log file
mem-pool logs filling up client log file
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: logging (Show other bugs)
mainline
Unspecified Unspecified
medium Severity unspecified
: ---
: ---
Assigned To: Amar Tumballi
Ujjwala
:
Depends On:
Blocks: 817967
  Show dependency treegraph
 
Reported: 2012-04-02 06:10 EDT by Anush Shetty
Modified: 2013-12-18 19:07 EST (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-24 13:44:44 EDT
Type: ---
Regression: ---
Mount Type: fuse
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Anush Shetty 2012-04-02 06:10:11 EDT
Description of problem: The fuse client log file in DEBUG mode is filled with log messages when mem pool is full


Version-Release number of selected component (if applicable): upstream

  
Actual results:

[2012-04-02 15:31:38.924422] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste
r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c
2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924478] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste
r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c
2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924533] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste
r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924588] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924644] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924699] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.924755] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.925128] D [client3_1-fops.c:2758:client_fdctx_destroy] 0-test2-client-0: sending releasedir on fd
[2012-04-02 15:31:38.928907] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.928999] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem
[2012-04-02 15:31:38.929057] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem


Expected results: Shouldn't be filling up so much. 


Additional info:
Comment 1 Amar Tumballi 2012-04-02 06:29:22 EDT
Anush, Is it possible to take a 'statedump' (with kill -USR1) of the process when this happens? I want the numbers of mempool structure variables to set to right values.
Comment 2 Anush Shetty 2012-04-02 07:28:15 EDT
pool-name=glusterfs:dict_t
hot-count=29
cold-count=995
padded_sizeof=84
alloc-count=314959
max-alloc=1024
pool-misses=82258
max-stdalloc=2548
Comment 3 Amar Tumballi 2012-04-11 14:14:48 EDT
looks like if we increase the dict_t pool count to 4096, we would not hit these logs so frequently. will send out a patch for this.
Comment 4 Anand Avati 2012-04-11 19:48:32 EDT
CHANGE: http://review.gluster.com/3131 (glusterfsd: mem-pool count of dict_t is adjusted) merged in master by Anand Avati (avati@redhat.com)
Comment 5 Anush Shetty 2012-05-23 08:08:07 EDT
Verified with 3.3.0qa42

Note You need to log in before you can comment on or make changes to this bug.