Bug 1349276

Summary:	Buffer overflow when attempting to create filesystem using libgfapi as driver on OpenStack
Product:	[Community] GlusterFS	Reporter:	Jiffin <jthottan>
Component:	libgfapi	Assignee:	Jiffin <jthottan>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Sudhir D <sdharane>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, joe, ndevos, rhbugzilla, sdharane
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.9.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1348935
Clones:	1350789 1350880 (view as bug list)		Environment:
Last Closed:	2017-03-27 18:14:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1348935, 1350789, 1350880

Description Jiffin 2016-06-23 06:46:03 UTC

+++ This bug was initially created as a clone of Bug #1348935 +++

Description of problem:

Having GlusterFS to provide OpenStack Cinder volume storage using libgfapi causes buffer overflow when trying to create a filesystem to attached volume. This results qemu-kvm process for the instance to be terminated.

Version-Release number of selected component (if applicable):
* GlusterFS 3.8.0 on all the involved servers
* CentOS 7.1
* libvirt-daemon-1.2.8-16.el7_1.5.x86_64
* qemu-kvm-1.5.3-86.el7_1.8.x86_64

How reproducible:
Tested in three different environments and all fail similarly.

Steps to Reproduce:
1. deploy an instance
2. attach volume (of type glusterfs)
3. attempt mkfs.ext4 /dev/vdb1

Actual results:
[2016-06-22 09:15:09.350992] E [glfs-fops.c:806:glfs_io_async_cbk] (-->/usr/lib64/glusterfs/3.8.0/xlator/debug/io-stats.so(+0x11e12) [0x7eff84cb8e12] -->/lib64/libgfapi.so.0(+0xbe7d) [0x7f0000ce2e7d] -->/lib64/libgfapi.so.0(+0xbd96) [0x7f0000ce2d96] ) 0-gfapi: invalid argument: iovec [Invalid argument]
*** buffer overflow detected ***: /usr/libexec/qemu-kvm terminated

Expected results:
* filesystem to be created without crashing the instance

Additional info:
There was no such issue with 3.7.11 but we upgraded due to memory leak issues with libgfapi.

--- Additional comment from Joe Julian on 2016-06-22 10:36:58 EDT ---

Unless I'm reading this wrong, every place that glfs_io_async_cbk is called, the return value is never checked so when that error takes place, none of the unrefs or frees are ever done.

Comment 1 Vijay Bellur 2016-06-23 07:03:19 UTC

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#1) for review on master by jiffin tony Thottan (jthottan)

Comment 2 Vijay Bellur 2016-06-23 07:19:29 UTC

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#2) for review on master by jiffin tony Thottan (jthottan)

Comment 3 Vijay Bellur 2016-06-27 11:49:21 UTC

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#3) for review on master by jiffin tony Thottan (jthottan)

Comment 4 Vijay Bellur 2016-06-27 11:52:45 UTC

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#4) for review on master by jiffin tony Thottan (jthottan)

Comment 5 Vijay Bellur 2016-06-28 11:21:59 UTC

COMMIT: http://review.gluster.org/14779 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit 61d72b3d91f2655b04de4ef29262f738a8cf7369
Author: Jiffin Tony Thottan <jthottan>
Date:   Thu Jun 23 12:20:03 2016 +0530

    gfapi : check the value "iovec" in glfs_io_async_cbk only for read
    
    The glfs_io_async_cbk() is called from the cbk of all the async ops
    such as write, read, fsync, ftruncate. In all other cases, expect for
    read the value for "iovec" is NULL. From the code, glfs_io_async_cbk
    checks the value in common routine which may end up in failures.
    
    Thanks Joe Julian for finding issue and suggesting the fix.
    
    Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5
    BUG: 1349276
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reported-by: Joe Julian <me>
    Reviewed-on: http://review.gluster.org/14779
    Reviewed-by: Niels de Vos <ndevos>
    Smoke: Gluster Build System <jenkins.org>
    Tested-by: Kaleb KEITHLEY <kkeithle>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Joe Julian <me>

Comment 6 Shyamsundar 2017-03-27 18:14:00 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report.

glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html
[2] https://www.gluster.org/pipermail/gluster-users/