Bug 1350880

Summary:	Buffer overflow when attempting to create filesystem using libgfapi as driver on OpenStack
Product:	[Community] GlusterFS	Reporter:	Niels de Vos <ndevos>
Component:	libgfapi	Assignee:	Niels de Vos <ndevos>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Sudhir D <sdharane>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	3.7.12	CC:	bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.13	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1349276	Environment:
Last Closed:	2016-07-20 13:55:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1349276
Bug Blocks:	1350804

Description Niels de Vos 2016-06-28 15:05:18 UTC

+++ This bug was initially created as a clone of Bug #1349276 +++

+++ This bug was initially created as a clone of Bug #1348935 +++

Description of problem:

Having GlusterFS to provide OpenStack Cinder volume storage using libgfapi causes buffer overflow when trying to create a filesystem to attached volume. This results qemu-kvm process for the instance to be terminated.

Version-Release number of selected component (if applicable):
* GlusterFS 3.8.0 on all the involved servers
* CentOS 7.1
* libvirt-daemon-1.2.8-16.el7_1.5.x86_64
* qemu-kvm-1.5.3-86.el7_1.8.x86_64

How reproducible:
Tested in three different environments and all fail similarly.

Steps to Reproduce:
1. deploy an instance
2. attach volume (of type glusterfs)
3. attempt mkfs.ext4 /dev/vdb1

Actual results:
[2016-06-22 09:15:09.350992] E [glfs-fops.c:806:glfs_io_async_cbk] (-->/usr/lib64/glusterfs/3.8.0/xlator/debug/io-stats.so(+0x11e12) [0x7eff84cb8e12] -->/lib64/libgfapi.so.0(+0xbe7d) [0x7f0000ce2e7d] -->/lib64/libgfapi.so.0(+0xbd96) [0x7f0000ce2d96] ) 0-gfapi: invalid argument: iovec [Invalid argument]
*** buffer overflow detected ***: /usr/libexec/qemu-kvm terminated

Expected results:
* filesystem to be created without crashing the instance

Additional info:
There was no such issue with 3.7.11 but we upgraded due to memory leak issues with libgfapi.

--- Additional comment from Joe Julian on 2016-06-22 10:36:58 EDT ---

Unless I'm reading this wrong, every place that glfs_io_async_cbk is called, the return value is never checked so when that error takes place, none of the unrefs or frees are ever done.

--- Additional comment from Vijay Bellur on 2016-06-23 09:03:19 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#1) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-23 09:19:29 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#2) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-27 13:49:21 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#3) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-27 13:52:45 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#4) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-28 13:21:59 CEST ---

COMMIT: http://review.gluster.org/14779 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit 61d72b3d91f2655b04de4ef29262f738a8cf7369
Author: Jiffin Tony Thottan <jthottan>
Date:   Thu Jun 23 12:20:03 2016 +0530

    gfapi : check the value "iovec" in glfs_io_async_cbk only for read
    
    The glfs_io_async_cbk() is called from the cbk of all the async ops
    such as write, read, fsync, ftruncate. In all other cases, expect for
    read the value for "iovec" is NULL. From the code, glfs_io_async_cbk
    checks the value in common routine which may end up in failures.
    
    Thanks Joe Julian for finding issue and suggesting the fix.
    
    Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5
    BUG: 1349276
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reported-by: Joe Julian <me>
    Reviewed-on: http://review.gluster.org/14779
    Reviewed-by: Niels de Vos <ndevos>
    Smoke: Gluster Build System <jenkins.org>
    Tested-by: Kaleb KEITHLEY <kkeithle>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Joe Julian <me>

Comment 1 Vijay Bellur 2016-06-28 15:09:38 UTC

REVIEW: http://review.gluster.org/14822 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)

Comment 2 Vijay Bellur 2016-06-29 12:47:23 UTC

COMMIT: http://review.gluster.org/14822 committed in release-3.7 by Niels de Vos (ndevos) 
------
commit 82a5e6cdacd9310f04830c47fd22e0aa9b7b7251
Author: Jiffin Tony Thottan <jthottan>
Date:   Thu Jun 23 12:20:03 2016 +0530

    gfapi : check the value "iovec" in glfs_io_async_cbk only for read
    
    The glfs_io_async_cbk() is called from the cbk of all the async ops
    such as write, read, fsync, ftruncate. In all other cases, expect for
    read the value for "iovec" is NULL. From the code, glfs_io_async_cbk
    checks the value in common routine which may end up in failures.
    
    Thanks Joe Julian for finding issue and suggesting the fix.
    
    Cherry picked from commit 61d72b3d91f2655b04de4ef29262f738a8cf7369:
    > Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5
    > BUG: 1349276
    > Signed-off-by: Jiffin Tony Thottan <jthottan>
    > Reported-by: Joe Julian <me>
    > Reviewed-on: http://review.gluster.org/14779
    > Reviewed-by: Niels de Vos <ndevos>
    > Smoke: Gluster Build System <jenkins.org>
    > Tested-by: Kaleb KEITHLEY <kkeithle>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Joe Julian <me>
    
    Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5
    BUG: 1350880
    Reported-by: Joe Julian <me>
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/14822
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: jiffin tony Thottan <jthottan>

Comment 3 Kaushal 2016-07-20 13:55:32 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.13, please open a new bug report.

glusterfs-3.7.13 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-July/027604.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user