Bug 1246203

Summary:	iobuf_unref errors killing logging ("again")
Product:	[Community] GlusterFS	Reporter:	Markus Ueberall <ueberall>
Component:	read-ahead	Assignee:	Raghavendra G <rgowdapp>
Status:	CLOSED NEXTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.6.4	CC:	bugs, kaushal, pgurusid, pkarampu, rkavunga, srangana
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-23 12:41:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Markus Ueberall 2015-07-23 17:01:31 UTC

Description of problem:
After bringing up the two servers of a mirrored volume, a large number of the below errors can be seen in the log during self-healing. (cf. bug #1116514)

[2015-07-23 16:40:02.832727] E [iobuf.c:759:iobuf_unref] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f707192dda6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv_cached+0x2a7)[0x7f7067579087] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv+0x4a)[0x7f706757914a] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readv_resume+0x13f)[0x7f707193dd1f] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(call_resume+0x5e4)[0x7f7071955e94] ))))) 0-iobuf: invalid argument: iobuf
[2015-07-23 16:40:02.833061] E [iobuf.c:865:iobref_unref] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f707192dda6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv_cached+0x2af)[0x7f706757908f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv+0x4a)[0x7f706757914a] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readv_resume+0x13f)[0x7f707193dd1f] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(call_resume+0x5e4)[0x7f7071955e94] ))))) 0-iobuf: invalid argument: iobref


Version-Release number of selected component (if applicable):
v3.6.4 (on Ubuntu Linux 14.04.2 LTS) [NB packages in question needed to be rebuilt locally with cherry-picked hotfix http://review.gluster.org/#/c/11650/ due to SSL-related errors]

How reproducible:
Just by bringing up both servers (always during self-healing process)

Steps to Reproduce:
1. reboot the hosts involved

Actual results:
Errors pointing to invalid arguments

Expected results:
No such errors

Additional info:
glusertd/glusterfsd processes die rather frequently.

Comment 1 Shyamsundar 2015-07-23 17:32:51 UTC

This seems to have crept back into the 3.6 and 3.5 branch and is fixed in master as a part of this patch http://review.gluster.org/#/c/10206/

History:
--------
The code that originally fixed the issue reported in BZ #1116514 was reverted as a part of http://review.gluster.org/#/c/8064/ (in master) and later fixed again in master as a part of http://review.gluster.org/#/c/10206/

Looks like second the fix is not back ported to the 3.6 line.

Checked the 3.5 code base as well, the fix is required in this branch too.
  - Original fix for #1116514 reverted on 3.5 line with this patch, 
http://review.gluster.org/7605

Comment 2 Shyamsundar 2015-07-23 17:36:02 UTC

Assigning to Du based on MAINTAINERS file.

Comment 3 Kaushal 2016-08-23 12:41:05 UTC

This bug is being closed as GlusterFS-3.6 is nearing its End-Of-Life and only important security bugs will be fixed. This bug has been fixed in more recent GlusterFS releases. If you still face this bug with the newer GlusterFS versions, please open a new bug.