Bug 1116514

Summary:	iobuf_unref errors killing logging
Product:	[Community] GlusterFS	Reporter:	Joe Julian <joe>
Component:	core	Assignee:	Vijay Bellur <vbellur>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.4.4	CC:	bugs, gluster-bugs, kkeithle, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.5beta2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1288202 (view as bug list)		Environment:
Last Closed:	2015-04-13 07:13:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1110456

Description Joe Julian 2014-07-05 15:25:02 UTC

After starting the client, a rapid sequence of these errors results in the last entry never being fully written an the log never goes any further.

After a couple of days, this client is dying but I have no idea why as there's no log beyond where it halts.

[root@mysql1 glusterfs]# tail -n 5 /var/log/glusterfs/mnt-gluster-mysql1.log
[2014-07-05 15:00:02.238570] E [iobuf.c:733:iobuf_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobuf
[2014-07-05 15:00:02.238613] E [iobuf.c:828:iobref_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobref
[2014-07-05 15:00:02.243550] E [iobuf.c:733:iobuf_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobuf
[2014-07-05 15:00:02.243594] E [iobuf.c:828:iobref_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7f028b9e0492]))) 0-iobuf: invalid argument: iobref
[2014-07-05 15:00:02.295156] E [iobuf.c:733:iobuf_unref] (-->/usr/lib64/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x17d) [0x7f028b5bec3d] (-->/usr/lib64/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x166) [0x7f028b7d36c6] (-->/usr/lib64/glusterfs/3.4.4/xlator/pe[root@mysql1 glusterfs]# 

Version-Release number of selected component (if applicable):
3.4.4-2

Volume Name: mysql1
Type: Distributed-Replicate
Volume ID: 713e6220-7eb2-4d12-9cd7-fe800b9b741a
Status: Started
Number of Bricks: 4 x 3 = 12
Transport-type: tcp
Bricks:
Brick1: ewcs2:/var/spool/glusterfs/a_mysql1
Brick2: ewcs10:/data/glusterfs/mysql1/a/brick
Brick3: ewcs7:/data/glusterfs/mysql1/a/brick
Brick4: ewcs2:/var/spool/glusterfs/b_mysql1
Brick5: ewcs10:/data/glusterfs/mysql1/b/brick
Brick6: ewcs7:/data/glusterfs/mysql1/b/brick
Brick7: ewcs2:/var/spool/glusterfs/c_mysql1
Brick8: ewcs10:/data/glusterfs/mysql1/c/brick
Brick9: ewcs7:/data/glusterfs/mysql1/c/brick
Brick10: ewcs2:/data/glusterfs/mysql1/d/brick
Brick11: ewcs10:/data/glusterfs/mysql1/d/brick
Brick12: ewcs7:/data/glusterfs/mysql1/d/brick
Options Reconfigured:
server.allow-insecure: on
performance.write-behind: off
nfs.disable: on
performance.cache-size: 8MB
cluster.background-self-heal-count: 2
cluster.data-self-heal-algorithm: full

Comment 1 Anand Avati 2014-07-05 15:54:17 UTC

REVIEW: http://review.gluster.org/8242 (performance/quick-read: Check for iobuf and iobref before unref.) posted (#1) for review on release-3.4 by Vijay Bellur (vbellur)

Comment 2 Anand Avati 2014-07-08 11:52:24 UTC

COMMIT: http://review.gluster.org/8242 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit b3bf2cfaa8e932f02a10a42e0281290a7cbdc960
Author: Vijay Bellur <vbellur>
Date:   Sat Jul 5 21:19:01 2014 +0530

    performance/quick-read: Check for iobuf and iobref before unref.
    
    In the absence of this check, iobuf_unref() can cause a log flood
    for read operations.
    
    Change-Id: Ib8beef6e55b69ecfb566bf61a96215ef0d0d74dc
    BUG: 1116514
    Signed-off-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8242
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Humble Devassy Chirammal <humble.devassy>
    Reviewed-by: Prashanth Pai <ppai>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>