+++ This bug was initially created as a clone of Bug #1344714 +++ Description of problem: The struct callback_inode_arg is a variable used by upcall infra for populating upcall entries. This variable is allocated using GF_CALLOC in glfs_h_poll_cache_invalidation(), but it also freedin FSAL_GLUSTER using free. This will result in a crash when we try to remove a file from mount Version-Release number of selected component (if applicable): mainline How reproducible: Steps to Reproduce: 1. enable nfs-ganesha and export a volume 2. mount the volume 3. create a file and delete it from the mount Actual results: ganesha server crashes Expected results: should not crash Additional info: --- Additional comment from Vijay Bellur on 2016-06-10 09:17:53 EDT --- REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free() api) posted (#1) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-10 09:17:56 EDT --- REVIEW: http://review.gluster.org/14702 (libgfapi/upcall : appending glfs to callback_arg, callback_inode_arg) posted (#1) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-10 16:57:16 EDT --- REVIEW: http://review.gluster.org/14702 (libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg) posted (#2) for review on master by Jeff Darcy (jdarcy) --- Additional comment from Vijay Bellur on 2016-06-10 17:09:37 EDT --- COMMIT: http://review.gluster.org/14702 committed in master by Jeff Darcy (jdarcy) ------ commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6 Author: Jiffin Tony Thottan <jthottan> Date: Fri Jun 10 18:23:43 2016 +0530 libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg Change-Id: I371525775db4f6a4d69beb94baaa53d17b16fb41 BUG: 1344714 Signed-off-by: Jiffin Tony Thottan <jthottan> Reviewed-on: http://review.gluster.org/14702 CentOS-regression: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Jeff Darcy <jdarcy> Tested-by: Jeff Darcy <jdarcy> Smoke: Gluster Build System <jenkins.com> --- Additional comment from Vijay Bellur on 2016-06-14 04:58:05 EDT --- REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#2) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-14 06:14:16 EDT --- REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#3) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-16 09:28:10 EDT --- REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#4) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-16 09:30:18 EDT --- REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#5) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-16 09:41:28 EDT --- REVIEW: http://review.gluster.org/14701 (libgfapi : free glfs_callback_inode_arg from application layer) posted (#6) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-17 08:46:07 EDT --- REVIEW: http://review.gluster.org/14759 (libgfapi/upcall : use calloc and free for callback_inode_arg) posted (#1) for review on master by jiffin tony Thottan (jthottan)
REVIEW: http://review.gluster.org/14760 (libgfapi/upcall : use calloc and free for callback_inode_arg) posted (#1) for review on release-3.7 by jiffin tony Thottan (jthottan)
REVIEW: http://review.gluster.org/15600 (Upcall/cache-invalidation: Use parent stbuf while updating parent entry) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/15601 (libglusterfs: add gf_get_mem_type()) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface for upcall consumers) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/15601 (libglusterfs: add gf_get_mem_type()) posted (#2) for review on release-3.7 by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface for upcall consumers) posted (#2) for review on release-3.7 by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/15600 committed in release-3.7 by Kaushal M (kaushal) ------ commit fee92f7c6abc651e88a8b52d5187286a4b8df2c3 Author: Soumya Koduri <skoduri> Date: Mon Dec 14 17:52:37 2015 +0530 Upcall/cache-invalidation: Use parent stbuf while updating parent entry For *create* fops (CREATE, MKDIR, MKNOD), we invalidate the parent entry. Hence send parent attributes in the stat field. Also "UP_PARENT_DENTRY_FLAGS" has to be set only for the fops which shall result in two invalidations requests - one for the inode on which fop is being performed and another on parent entry. In case of CREATE/MKDIR/MKNOD fops, there shall be only one invalidation request sent, that too on parent inode. We send invalidation directly on parent inode's gfid. So there is no necessity to set these flags which when set shall endup invalidating the parent's parent entry. Cherry picked from commit f4282bd927e2e0d826d62cf1192102382c5697b2: > Change-Id: I7514ee08382081e3e060818ede497dbca26987dc > BUG: 1291259 > Signed-off-by: Soumya Koduri <skoduri> > Reviewed-on: http://review.gluster.org/12962 > CentOS-regression: Gluster Build System <jenkins.com> > NetBSD-regression: NetBSD Build System <jenkins.org> > Smoke: Gluster Build System <jenkins.com> > Reviewed-by: Niels de Vos <ndevos> Change-Id: I7514ee08382081e3e060818ede497dbca26987dc BUG: 1347715 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/15600 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Kaushal M <kaushal>
COMMIT: http://review.gluster.org/15601 committed in release-3.7 by Kaushal M (kaushal) ------ commit 2cee4a3e47518aeb28ac2b611c6f01c0f9d00dab Author: Niels de Vos <ndevos> Date: Sat Aug 6 16:04:29 2016 +0200 libglusterfs: add gf_get_mem_type() gfapi needs to provide a function towards applications to free memory that it allocated. Depending on how the application is compiled/linked, it could use a different memory allocator than Gluster itself. Therefore it is not safe for gfapi to request applications to free memory with 'standard' free(). Examples for this are Gluster allocated structures with GF_CALLOC() when memory accounting is enabled (the default). Some gfapi functions use malloc() to allocate memory as a workaround, but the free() from the jemalloc implementation should not be combined with the malloc() from glibc. Cherry picked from commit db4e26ed71a01e5f760fbc3c7051962426f102c9: > Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093 > BUG: 1344714 > Signed-off-by: Niels de Vos <ndevos> > Reviewed-on: http://review.gluster.org/15108 > Smoke: Gluster Build System <jenkins.org> > Reviewed-by: soumya k <skoduri> > Reviewed-by: jiffin tony Thottan <jthottan> > NetBSD-regression: NetBSD Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> > Reviewed-by: Shyamsundar Ranganathan <srangana> Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093 BUG: 1347715 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/15601 NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Kaushal M <kaushal>
REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface for upcall consumers) posted (#3) for review on release-3.7 by Kaushal M (kaushal)
COMMIT: http://review.gluster.org/15602 committed in release-3.7 by Kaushal M (kaushal) ------ commit 55c92db32ba7d88359f0562953a3a6d8874dd1a5 Author: Niels de Vos <ndevos> Date: Fri Jun 10 18:23:43 2016 +0530 gfapi: redesign the public interface for upcall consumers The glfs_callback_arg and glfs_callback_inode_arg were allocated by gfapi, and expected to be free()'d by the application. However it is not reasonable to expect that applications use the same memory allocator to as the compiled libgfapi.so. For instance, it is possible that gfapi uses glibc malloc/free, and an application like NFS-Ganesha the versions from jemalloc. Mismatching of the malloc() and free() functions causes segmentation faults at best. In order to prevent problems like this in the future, the API for applications that consume upcalls has been remodeled. Any of the structures that gfapi allocates, should be free'd with glfs_free(). The members of the structures can not be accessed directly anymore, each has its own function to access now. Correcting the naming of the functions, structures and constants is a continuation of commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6. These new improvements not only have correct prefixes for the functions and structures, the naming also reflects more to the upcall framework and does not use "callback" anymore. Cherry picked from commit 4721188a154acd9a0a4c096d8d73e97f3bf1b2a9: > Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93 > BUG: 1344714 > Signed-off-by: Niels de Vos <ndevos> > Reviewed-on: http://review.gluster.org/14701 > Smoke: Gluster Build System <jenkins.org> > NetBSD-regression: NetBSD Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> > Reviewed-by: Kaleb KEITHLEY <kkeithle> > Reviewed-by: soumya k <skoduri> > Reviewed-by: jiffin tony Thottan <jthottan> Once difference with the version of this change in other branches is that leases are not included in glusterfs-3.7. Hence there is a little change that drops the handling of GF_UPCALL_RECALL_LEASE. In addition, this backport contains commit 2775dc6410: > libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg > Reviewed-on: http://review.gluster.org/14702 Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93 BUG: 1347715 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/15602 NetBSD-regression: NetBSD Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Kaushal M <kaushal>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.16, please open a new bug report. glusterfs-3.7.16 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://www.gluster.org/pipermail/gluster-devel/2016-October/051187.html [2] https://www.gluster.org/pipermail/gluster-users/