Bug 1350880 - Buffer overflow when attempting to create filesystem using libgfapi as driver on OpenStack
Summary: Buffer overflow when attempting to create filesystem using libgfapi as driver...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: libgfapi
Version: 3.7.12
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
Assignee: Niels de Vos
QA Contact: Sudhir D
URL:
Whiteboard:
Depends On: 1349276
Blocks: glusterfs-3.7.13
TreeView+ depends on / blocked
 
Reported: 2016-06-28 15:05 UTC by Niels de Vos
Modified: 2016-07-20 13:55 UTC (History)
1 user (show)

Fixed In Version: glusterfs-3.7.13
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1349276
Environment:
Last Closed: 2016-07-20 13:55:32 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1333268 0 unspecified CLOSED SMB:while running I/O on cifs mount and doing graph switch causes cifs mount to hang. 2021-02-22 00:41:40 UTC

Internal Links: 1333268

Description Niels de Vos 2016-06-28 15:05:18 UTC
+++ This bug was initially created as a clone of Bug #1349276 +++

+++ This bug was initially created as a clone of Bug #1348935 +++

Description of problem:

Having GlusterFS to provide OpenStack Cinder volume storage using libgfapi causes buffer overflow when trying to create a filesystem to attached volume. This results qemu-kvm process for the instance to be terminated.

Version-Release number of selected component (if applicable):
* GlusterFS 3.8.0 on all the involved servers
* CentOS 7.1
* libvirt-daemon-1.2.8-16.el7_1.5.x86_64
* qemu-kvm-1.5.3-86.el7_1.8.x86_64

How reproducible:
Tested in three different environments and all fail similarly.

Steps to Reproduce:
1. deploy an instance
2. attach volume (of type glusterfs)
3. attempt mkfs.ext4 /dev/vdb1

Actual results:
[2016-06-22 09:15:09.350992] E [glfs-fops.c:806:glfs_io_async_cbk] (-->/usr/lib64/glusterfs/3.8.0/xlator/debug/io-stats.so(+0x11e12) [0x7eff84cb8e12] -->/lib64/libgfapi.so.0(+0xbe7d) [0x7f0000ce2e7d] -->/lib64/libgfapi.so.0(+0xbd96) [0x7f0000ce2d96] ) 0-gfapi: invalid argument: iovec [Invalid argument]
*** buffer overflow detected ***: /usr/libexec/qemu-kvm terminated

Expected results:
* filesystem to be created without crashing the instance

Additional info:
There was no such issue with 3.7.11 but we upgraded due to memory leak issues with libgfapi.

--- Additional comment from Joe Julian on 2016-06-22 10:36:58 EDT ---

Unless I'm reading this wrong, every place that glfs_io_async_cbk is called, the return value is never checked so when that error takes place, none of the unrefs or frees are ever done.

--- Additional comment from Vijay Bellur on 2016-06-23 09:03:19 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#1) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-23 09:19:29 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#2) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-27 13:49:21 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#3) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-27 13:52:45 CEST ---

REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#4) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-28 13:21:59 CEST ---

COMMIT: http://review.gluster.org/14779 committed in master by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit 61d72b3d91f2655b04de4ef29262f738a8cf7369
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Thu Jun 23 12:20:03 2016 +0530

    gfapi : check the value "iovec" in glfs_io_async_cbk only for read
    
    The glfs_io_async_cbk() is called from the cbk of all the async ops
    such as write, read, fsync, ftruncate. In all other cases, expect for
    read the value for "iovec" is NULL. From the code, glfs_io_async_cbk
    checks the value in common routine which may end up in failures.
    
    Thanks Joe Julian for finding issue and suggesting the fix.
    
    Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5
    BUG: 1349276
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reported-by: Joe Julian <me@joejulian.name>
    Reviewed-on: http://review.gluster.org/14779
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Joe Julian <me@joejulian.name>

Comment 1 Vijay Bellur 2016-06-28 15:09:38 UTC
REVIEW: http://review.gluster.org/14822 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#1) for review on release-3.7 by Niels de Vos (ndevos@redhat.com)

Comment 2 Vijay Bellur 2016-06-29 12:47:23 UTC
COMMIT: http://review.gluster.org/14822 committed in release-3.7 by Niels de Vos (ndevos@redhat.com) 
------
commit 82a5e6cdacd9310f04830c47fd22e0aa9b7b7251
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Thu Jun 23 12:20:03 2016 +0530

    gfapi : check the value "iovec" in glfs_io_async_cbk only for read
    
    The glfs_io_async_cbk() is called from the cbk of all the async ops
    such as write, read, fsync, ftruncate. In all other cases, expect for
    read the value for "iovec" is NULL. From the code, glfs_io_async_cbk
    checks the value in common routine which may end up in failures.
    
    Thanks Joe Julian for finding issue and suggesting the fix.
    
    Cherry picked from commit 61d72b3d91f2655b04de4ef29262f738a8cf7369:
    > Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5
    > BUG: 1349276
    > Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    > Reported-by: Joe Julian <me@joejulian.name>
    > Reviewed-on: http://review.gluster.org/14779
    > Reviewed-by: Niels de Vos <ndevos@redhat.com>
    > Smoke: Gluster Build System <jenkins@build.gluster.org>
    > Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    > Reviewed-by: Joe Julian <me@joejulian.name>
    
    Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5
    BUG: 1350880
    Reported-by: Joe Julian <me@joejulian.name>
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: http://review.gluster.org/14822
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: jiffin tony Thottan <jthottan@redhat.com>

Comment 3 Kaushal 2016-07-20 13:55:32 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.13, please open a new bug report.

glusterfs-3.7.13 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-July/027604.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.