Description of problem: Got lots of this from syslog: Apr 23 13:57:41 localhost GlusterFS[16976]: [2013-04-23 01:57:41.505530] C [mem-pool.c:497:mem_put] (-->/usr/lib/glusterfs/3.4.0alpha2/xlator/performance/io-cache.so(ioc_readv+0x3ab) [0x7f5090b2f34b] (-->/usr/lib/glusterfs/3.4. 0alpha2/xlator/performance/io-cache.so(ioc_dispatch_requests+0x3a8) [0x7f5090b2ee88] (-->/usr/lib/glusterfs/3.4.0alpha2/xlator/performance/io-cache.so(ioc_frame_return+0x3c1) [0x7f5090b31c71]))) 0-mem-pool: mem_put called on fr eed ptr 0x1fbc874 of mem pool 0x1fbaa30 and /usr/sbin/glusterfsd for that volume stop worknig how to debug and solve this problem? When running all seems to be fine. gluster volume info Volume Name: VM Type: Replicate Volume ID: 72cf884d-83a4-459c-8601-b7b3e6ef7308 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vhoc1h:/mnt/kvmimages/data Brick2: vhoc2h:/mnt/kvmimages/data Brick3: vhoc3h:/mnt/kvmimages/data when fail: gluster> volume status Status of volume: VM Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick vhoc2h:/mnt/kvmimages/data N/A N 4758 Brick vhoc3h:/mnt/kvmimages/data N/A N 16718 NFS Server on localhost 38467 Y 16758 Self-heal Daemon on localhost N/A Y 16769 NFS Server on c2e876af-784e-4b29-95dd-50f2d8d69ab8 38467 Y 4864 Self-heal Daemon on c2e876af-784e-4b29-95dd-50f2d8d69ab 8 N/A Y 4871 There are no active volume tasks Status of volume: VMCOL Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick vhoc2h:/mnt/kvmcollege/data 47154 Y 4765 Brick vhoc3h:/mnt/kvmcollege/data 47154 Y 16748 NFS Server on localhost 38467 Y 16758 Self-heal Daemon on localhost N/A Y 16769 NFS Server on c2e876af-784e-4b29-95dd-50f2d8d69ab8 38467 Y 4864 Self-heal Daemon on c2e876af-784e-4b29-95dd-50f2d8d69ab 8 N/A Y 4871 There are no active volume tasks and from /var/log/glusterfs/bricks/mnt-kvmimages-data.log : [2013-04-22 20:02:44.758083] I [glusterfsd-mgmt.c:1583:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing pending frames: frame : type(0) op(34) frame : type(0) op(34) frame : type(0) op(34) frame : type(0) op(34) ....... repeated 633 times Version-Release number of selected component (if applicable): gluster3.4alpha2 ( 3.4.alpha 3 failing at self-heal daemon ) How reproducible: Do not know. Steps to Reproduce: 1. 2. 3. Actual results: glusterfd stop vorking and mounted volume stop working as well Expected results: mounted volume continue to run from other nodes Additional info: Build from sources on debian wheezy
Steps to recreate: 1) When reads are in progress kill the bricks from which the reads are happening In the error handling code path of io-cache, the issue is seen.
REVIEW: http://review.gluster.org/4884 (performance/io-cache: Avoid double mem_put in ioc_readv) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/4886 (performance/io-cache: Avoid double mem_put in ioc_readv) posted (#1) for review on release-3.4 by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/4884 committed in master by Vijay Bellur (vbellur) ------ commit 6419bd804b9cf43ad08fc54ef605c77b0e7895af Author: Pranith Kumar K <pkarampu> Date: Wed Apr 24 18:05:13 2013 +0530 performance/io-cache: Avoid double mem_put in ioc_readv On readv error io-cache frame->local is not set to NULL so the local is mem_put in STACK_DESTROY as well. This patch sets frame->local to NULL in all cases. Change-Id: I00013df1377475aa5f3c0c681dcb58b32e1e8063 BUG: 955751 Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/4884 Reviewed-by: Raghavendra G <raghavendra> Tested-by: Gluster Build System <jenkins.com>
COMMIT: http://review.gluster.org/4886 committed in release-3.4 by Vijay Bellur (vbellur) ------ commit 5925d8a2512b8d4452b0b0efbafd9c7536ed3a51 Author: Pranith Kumar K <pkarampu> Date: Wed Apr 24 18:05:13 2013 +0530 performance/io-cache: Avoid double mem_put in ioc_readv On readv error io-cache frame->local is not set to NULL so the local is mem_put in STACK_DESTROY as well. This patch sets frame->local to NULL in all cases. BUG: 955751 Change-Id: I4a7340189efe02473452986b5870b02fcfa9038e Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/4886 Reviewed-by: Raghavendra G <raghavendra> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>