+++ This bug was initially created as a clone of Bug #1349276 +++ +++ This bug was initially created as a clone of Bug #1348935 +++ Description of problem: Having GlusterFS to provide OpenStack Cinder volume storage using libgfapi causes buffer overflow when trying to create a filesystem to attached volume. This results qemu-kvm process for the instance to be terminated. Version-Release number of selected component (if applicable): * GlusterFS 3.8.0 on all the involved servers * CentOS 7.1 * libvirt-daemon-1.2.8-16.el7_1.5.x86_64 * qemu-kvm-1.5.3-86.el7_1.8.x86_64 How reproducible: Tested in three different environments and all fail similarly. Steps to Reproduce: 1. deploy an instance 2. attach volume (of type glusterfs) 3. attempt mkfs.ext4 /dev/vdb1 Actual results: [2016-06-22 09:15:09.350992] E [glfs-fops.c:806:glfs_io_async_cbk] (-->/usr/lib64/glusterfs/3.8.0/xlator/debug/io-stats.so(+0x11e12) [0x7eff84cb8e12] -->/lib64/libgfapi.so.0(+0xbe7d) [0x7f0000ce2e7d] -->/lib64/libgfapi.so.0(+0xbd96) [0x7f0000ce2d96] ) 0-gfapi: invalid argument: iovec [Invalid argument] *** buffer overflow detected ***: /usr/libexec/qemu-kvm terminated Expected results: * filesystem to be created without crashing the instance Additional info: There was no such issue with 3.7.11 but we upgraded due to memory leak issues with libgfapi. --- Additional comment from Joe Julian on 2016-06-22 10:36:58 EDT --- Unless I'm reading this wrong, every place that glfs_io_async_cbk is called, the return value is never checked so when that error takes place, none of the unrefs or frees are ever done. --- Additional comment from Vijay Bellur on 2016-06-23 09:03:19 CEST --- REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#1) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-23 09:19:29 CEST --- REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#2) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-27 13:49:21 CEST --- REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#3) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-27 13:52:45 CEST --- REVIEW: http://review.gluster.org/14779 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#4) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-28 13:21:59 CEST --- COMMIT: http://review.gluster.org/14779 committed in master by Kaleb KEITHLEY (kkeithle) ------ commit 61d72b3d91f2655b04de4ef29262f738a8cf7369 Author: Jiffin Tony Thottan <jthottan> Date: Thu Jun 23 12:20:03 2016 +0530 gfapi : check the value "iovec" in glfs_io_async_cbk only for read The glfs_io_async_cbk() is called from the cbk of all the async ops such as write, read, fsync, ftruncate. In all other cases, expect for read the value for "iovec" is NULL. From the code, glfs_io_async_cbk checks the value in common routine which may end up in failures. Thanks Joe Julian for finding issue and suggesting the fix. Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5 BUG: 1349276 Signed-off-by: Jiffin Tony Thottan <jthottan> Reported-by: Joe Julian <me> Reviewed-on: http://review.gluster.org/14779 Reviewed-by: Niels de Vos <ndevos> Smoke: Gluster Build System <jenkins.org> Tested-by: Kaleb KEITHLEY <kkeithle> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Joe Julian <me>
REVIEW: http://review.gluster.org/14822 (gfapi : check the value "iovec" in glfs_io_async_cbk only for read) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/14822 committed in release-3.7 by Niels de Vos (ndevos) ------ commit 82a5e6cdacd9310f04830c47fd22e0aa9b7b7251 Author: Jiffin Tony Thottan <jthottan> Date: Thu Jun 23 12:20:03 2016 +0530 gfapi : check the value "iovec" in glfs_io_async_cbk only for read The glfs_io_async_cbk() is called from the cbk of all the async ops such as write, read, fsync, ftruncate. In all other cases, expect for read the value for "iovec" is NULL. From the code, glfs_io_async_cbk checks the value in common routine which may end up in failures. Thanks Joe Julian for finding issue and suggesting the fix. Cherry picked from commit 61d72b3d91f2655b04de4ef29262f738a8cf7369: > Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5 > BUG: 1349276 > Signed-off-by: Jiffin Tony Thottan <jthottan> > Reported-by: Joe Julian <me> > Reviewed-on: http://review.gluster.org/14779 > Reviewed-by: Niels de Vos <ndevos> > Smoke: Gluster Build System <jenkins.org> > Tested-by: Kaleb KEITHLEY <kkeithle> > NetBSD-regression: NetBSD Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> > Reviewed-by: Joe Julian <me> Change-Id: I0be0123da68f9d8fbb5d94ede2d45566a9add6a5 BUG: 1350880 Reported-by: Joe Julian <me> Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/14822 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Kaleb KEITHLEY <kkeithle> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: jiffin tony Thottan <jthottan>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.13, please open a new bug report. glusterfs-3.7.13 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://www.gluster.org/pipermail/gluster-users/2016-July/027604.html [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user