Description of problem: The fuse client log file in DEBUG mode is filled with log messages when mem pool is full Version-Release number of selected component (if applicable): upstream Actual results: [2012-04-02 15:31:38.924422] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c 2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.924478] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c 2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.924533] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluste r/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.924588] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.924644] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.924699] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.924755] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.925128] D [client3_1-fops.c:2758:client_fdctx_destroy] 0-test2-client-0: sending releasedir on fd [2012-04-02 15:31:38.928907] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.928999] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem [2012-04-02 15:31:38.929057] D [mem-pool.c:466:mem_get] (-->/gluster/review/lib/libglusterfs.so.0(dict_new+0x1a) [0x7f2d045deded] (-->/gluster/review/lib/libglusterfs.so.0(get_new_dict_full+0x2e) [0x7f2d045ded41] (-->/gluster/review/lib/libglusterfs.so.0(mem_get0+0x7b) [0x7f2d0461c2d6]))) 0-mem-pool: Mem pool is full. Callocing mem Expected results: Shouldn't be filling up so much. Additional info:
Anush, Is it possible to take a 'statedump' (with kill -USR1) of the process when this happens? I want the numbers of mempool structure variables to set to right values.
pool-name=glusterfs:dict_t hot-count=29 cold-count=995 padded_sizeof=84 alloc-count=314959 max-alloc=1024 pool-misses=82258 max-stdalloc=2548
looks like if we increase the dict_t pool count to 4096, we would not hit these logs so frequently. will send out a patch for this.
CHANGE: http://review.gluster.com/3131 (glusterfsd: mem-pool count of dict_t is adjusted) merged in master by Anand Avati (avati)
Verified with 3.3.0qa42