Bug 1246203 - iobuf_unref errors killing logging ("again")
Summary: iobuf_unref errors killing logging ("again")
Alias: None
Product: GlusterFS
Classification: Community
Component: read-ahead
Version: 3.6.4
Hardware: x86_64
OS: Linux
Target Milestone: ---
Assignee: Raghavendra G
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2015-07-23 17:01 UTC by Markus Ueberall
Modified: 2016-08-23 12:41 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-08-23 12:41:05 UTC
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:

Attachments (Terms of Use)

Description Markus Ueberall 2015-07-23 17:01:31 UTC
Description of problem:
After bringing up the two servers of a mirrored volume, a large number of the below errors can be seen in the log during self-healing. (cf. bug #1116514)

[2015-07-23 16:40:02.832727] E [iobuf.c:759:iobuf_unref] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f707192dda6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv_cached+0x2a7)[0x7f7067579087] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv+0x4a)[0x7f706757914a] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readv_resume+0x13f)[0x7f707193dd1f] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(call_resume+0x5e4)[0x7f7071955e94] ))))) 0-iobuf: invalid argument: iobuf
[2015-07-23 16:40:02.833061] E [iobuf.c:865:iobref_unref] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f707192dda6] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv_cached+0x2af)[0x7f706757908f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.6.4/xlator/performance/quick-read.so(qr_readv+0x4a)[0x7f706757914a] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readv_resume+0x13f)[0x7f707193dd1f] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(call_resume+0x5e4)[0x7f7071955e94] ))))) 0-iobuf: invalid argument: iobref

Version-Release number of selected component (if applicable):
v3.6.4 (on Ubuntu Linux 14.04.2 LTS) [NB packages in question needed to be rebuilt locally with cherry-picked hotfix http://review.gluster.org/#/c/11650/ due to SSL-related errors]

How reproducible:
Just by bringing up both servers (always during self-healing process)

Steps to Reproduce:
1. reboot the hosts involved

Actual results:
Errors pointing to invalid arguments

Expected results:
No such errors

Additional info:
glusertd/glusterfsd processes die rather frequently.

Comment 1 Shyamsundar 2015-07-23 17:32:51 UTC
This seems to have crept back into the 3.6 and 3.5 branch and is fixed in master as a part of this patch http://review.gluster.org/#/c/10206/

The code that originally fixed the issue reported in BZ #1116514 was reverted as a part of http://review.gluster.org/#/c/8064/ (in master) and later fixed again in master as a part of http://review.gluster.org/#/c/10206/

Looks like second the fix is not back ported to the 3.6 line.

Checked the 3.5 code base as well, the fix is required in this branch too.
  - Original fix for #1116514 reverted on 3.5 line with this patch, 

Comment 2 Shyamsundar 2015-07-23 17:36:02 UTC
Assigning to Du based on MAINTAINERS file.

Comment 3 Kaushal 2016-08-23 12:41:05 UTC
This bug is being closed as GlusterFS-3.6 is nearing its End-Of-Life and only important security bugs will be fixed. This bug has been fixed in more recent GlusterFS releases. If you still face this bug with the newer GlusterFS versions, please open a new bug.

Note You need to log in before you can comment on or make changes to this bug.