Bug 955751

Summary:	fuse mount crashes because of double mem_put in io-cache
Product:	[Community] GlusterFS	Reporter:	Michael <michael.auckland>
Component:	io-cache	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	pre-release	CC:	gluster-bugs, pkarampu, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-07-24 17:59:21 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	952693

Description Michael 2013-04-23 17:26:23 UTC

Description of problem:

Got lots of this from syslog:

Apr 23 13:57:41 localhost GlusterFS[16976]: [2013-04-23 01:57:41.505530] C [mem-pool.c:497:mem_put] (-->/usr/lib/glusterfs/3.4.0alpha2/xlator/performance/io-cache.so(ioc_readv+0x3ab) [0x7f5090b2f34b] (-->/usr/lib/glusterfs/3.4.
0alpha2/xlator/performance/io-cache.so(ioc_dispatch_requests+0x3a8) [0x7f5090b2ee88] (-->/usr/lib/glusterfs/3.4.0alpha2/xlator/performance/io-cache.so(ioc_frame_return+0x3c1) [0x7f5090b31c71]))) 0-mem-pool: mem_put called on fr
eed ptr 0x1fbc874 of mem pool 0x1fbaa30

and /usr/sbin/glusterfsd for that volume stop worknig
how to debug and solve this problem?

When running all seems to be fine.
gluster volume info
 
Volume Name: VM
Type: Replicate
Volume ID: 72cf884d-83a4-459c-8601-b7b3e6ef7308
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: vhoc1h:/mnt/kvmimages/data
Brick2: vhoc2h:/mnt/kvmimages/data
Brick3: vhoc3h:/mnt/kvmimages/data

when fail:

gluster> volume status
Status of volume: VM
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick vhoc2h:/mnt/kvmimages/data                        N/A     N       4758
Brick vhoc3h:/mnt/kvmimages/data                        N/A     N       16718
NFS Server on localhost                                 38467   Y       16758
Self-heal Daemon on localhost                           N/A     Y       16769
NFS Server on c2e876af-784e-4b29-95dd-50f2d8d69ab8      38467   Y       4864
Self-heal Daemon on c2e876af-784e-4b29-95dd-50f2d8d69ab
8                                                       N/A     Y       4871

There are no active volume tasks
Status of volume: VMCOL
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick vhoc2h:/mnt/kvmcollege/data                       47154   Y       4765
Brick vhoc3h:/mnt/kvmcollege/data                       47154   Y       16748
NFS Server on localhost                                 38467   Y       16758
Self-heal Daemon on localhost                           N/A     Y       16769
NFS Server on c2e876af-784e-4b29-95dd-50f2d8d69ab8      38467   Y       4864
Self-heal Daemon on c2e876af-784e-4b29-95dd-50f2d8d69ab
8                                                       N/A     Y       4871

There are no active volume tasks

and from /var/log/glusterfs/bricks/mnt-kvmimages-data.log :
[2013-04-22 20:02:44.758083] I [glusterfsd-mgmt.c:1583:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing                                              
pending frames:
frame : type(0) op(34)
frame : type(0) op(34)
frame : type(0) op(34)
frame : type(0) op(34)
....... repeated 633 times


Version-Release number of selected component (if applicable):
gluster3.4alpha2  ( 3.4.alpha 3 failing at self-heal daemon )

How reproducible:
Do not know.

Steps to Reproduce:
1.
2.
3.
  
Actual results:
glusterfd stop vorking and mounted volume stop working as well

Expected results:
mounted volume continue to run from other nodes

Additional info:
Build from sources on debian wheezy

Comment 1 Pranith Kumar K 2013-04-25 05:33:56 UTC

Steps to recreate:
1) When reads are in progress kill the bricks from which the reads are happening
In the error handling code path of io-cache, the issue is seen.

Comment 2 Anand Avati 2013-04-25 05:37:24 UTC

REVIEW: http://review.gluster.org/4884 (performance/io-cache: Avoid double mem_put in ioc_readv) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 3 Anand Avati 2013-04-25 05:44:30 UTC

REVIEW: http://review.gluster.org/4886 (performance/io-cache: Avoid double mem_put in ioc_readv) posted (#1) for review on release-3.4 by Pranith Kumar Karampuri (pkarampu)

Comment 4 Anand Avati 2013-04-26 07:19:22 UTC

COMMIT: http://review.gluster.org/4884 committed in master by Vijay Bellur (vbellur) 
------
commit 6419bd804b9cf43ad08fc54ef605c77b0e7895af
Author: Pranith Kumar K <pkarampu>
Date:   Wed Apr 24 18:05:13 2013 +0530

    performance/io-cache: Avoid double mem_put in ioc_readv
    
    On readv error io-cache frame->local is not set to NULL
    so the local is mem_put in STACK_DESTROY as well. This
    patch sets frame->local to NULL in all cases.
    
    Change-Id: I00013df1377475aa5f3c0c681dcb58b32e1e8063
    BUG: 955751
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/4884
    Reviewed-by: Raghavendra G <raghavendra>
    Tested-by: Gluster Build System <jenkins.com>

Comment 5 Anand Avati 2013-05-07 11:40:13 UTC

COMMIT: http://review.gluster.org/4886 committed in release-3.4 by Vijay Bellur (vbellur) 
------
commit 5925d8a2512b8d4452b0b0efbafd9c7536ed3a51
Author: Pranith Kumar K <pkarampu>
Date:   Wed Apr 24 18:05:13 2013 +0530

    performance/io-cache: Avoid double mem_put in ioc_readv
    
    On readv error io-cache frame->local is not set to NULL
    so the local is mem_put in STACK_DESTROY as well. This
    patch sets frame->local to NULL in all cases.
    
    BUG: 955751
    Change-Id: I4a7340189efe02473452986b5870b02fcfa9038e
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/4886
    Reviewed-by: Raghavendra G <raghavendra>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>