Bug 1391851

+++ This bug was initially created as a clone of Bug #1347715 +++

+++ This bug was initially created as a clone of Bug #1344714 +++

Description of problem:

The struct callback_inode_arg is a variable used by upcall infra for populating upcall entries. This variable is allocated using GF_CALLOC in glfs_h_poll_cache_invalidation(), but it also freedin FSAL_GLUSTER using free.
This will result in a crash when we try to remove a file from mount

Version-Release number of selected component (if applicable):
mainline

How reproducible:


Steps to Reproduce:
1. enable nfs-ganesha and export a volume 
2. mount the volume
3. create a file and delete it from the mount

Actual results:
ganesha server crashes

Expected results:
should not crash

Additional info:

--- Additional comment from Vijay Bellur on 2016-06-10 09:17:53 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free() api) posted (#1) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-10 09:17:56 EDT ---

REVIEW: http://review.gluster.org/14702 (libgfapi/upcall : appending glfs to callback_arg, callback_inode_arg) posted (#1) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-10 16:57:16 EDT ---

REVIEW: http://review.gluster.org/14702 (libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg) posted (#2) for review on master by Jeff Darcy (jdarcy)

--- Additional comment from Vijay Bellur on 2016-06-10 17:09:37 EDT ---

COMMIT: http://review.gluster.org/14702 committed in master by Jeff Darcy (jdarcy) 
------
commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6
Author: Jiffin Tony Thottan <jthottan>
Date:   Fri Jun 10 18:23:43 2016 +0530

    libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg
    
    Change-Id: I371525775db4f6a4d69beb94baaa53d17b16fb41
    BUG: 1344714
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14702
    CentOS-regression: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jdarcy>
    Tested-by: Jeff Darcy <jdarcy>
    Smoke: Gluster Build System <jenkins.com>

--- Additional comment from Vijay Bellur on 2016-06-14 04:58:05 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#2) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-14 06:14:16 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#3) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-16 09:28:10 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#4) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-16 09:30:18 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free_callback_inode_arg() api) posted (#5) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-16 09:41:28 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : free glfs_callback_inode_arg from application layer) posted (#6) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-17 08:46:07 EDT ---

REVIEW: http://review.gluster.org/14759 (libgfapi/upcall : use calloc and free for callback_inode_arg) posted (#1) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-17 08:55:22 EDT ---

REVIEW: http://review.gluster.org/14760 (libgfapi/upcall : use calloc and free for callback_inode_arg) posted (#1) for review on release-3.7 by jiffin tony Thottan (jthottan)

--- Additional comment from Worker Ant on 2016-09-30 06:48:37 EDT ---

REVIEW: http://review.gluster.org/15600 (Upcall/cache-invalidation: Use parent stbuf while updating parent entry) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)

--- Additional comment from Worker Ant on 2016-09-30 06:48:40 EDT ---

REVIEW: http://review.gluster.org/15601 (libglusterfs: add gf_get_mem_type()) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)

--- Additional comment from Worker Ant on 2016-09-30 06:48:43 EDT ---

REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface for upcall consumers) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)

--- Additional comment from Worker Ant on 2016-09-30 08:11:17 EDT ---

REVIEW: http://review.gluster.org/15601 (libglusterfs: add gf_get_mem_type()) posted (#2) for review on release-3.7 by Niels de Vos (ndevos)

--- Additional comment from Worker Ant on 2016-09-30 08:12:08 EDT ---

REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface for upcall consumers) posted (#2) for review on release-3.7 by Niels de Vos (ndevos)

--- Additional comment from Worker Ant on 2016-10-03 08:15:30 EDT ---

COMMIT: http://review.gluster.org/15600 committed in release-3.7 by Kaushal M (kaushal) 
------
commit fee92f7c6abc651e88a8b52d5187286a4b8df2c3
Author: Soumya Koduri <skoduri>
Date:   Mon Dec 14 17:52:37 2015 +0530

    Upcall/cache-invalidation: Use parent stbuf while updating parent entry
    
    For *create*  fops (CREATE, MKDIR, MKNOD), we invalidate the parent
    entry. Hence send parent attributes in the stat field.
    
    Also "UP_PARENT_DENTRY_FLAGS" has to be set only for the fops which shall
    result in two invalidations requests - one for the inode on which fop is
    being performed and another on parent entry.
    
    In case of CREATE/MKDIR/MKNOD fops, there shall be only one invalidation
    request sent, that too on parent inode. We send invalidation directly on
    parent inode's gfid. So there is no necessity to set these flags which
    when set shall endup invalidating the parent's parent entry.
    
    Cherry picked from commit f4282bd927e2e0d826d62cf1192102382c5697b2:
    > Change-Id: I7514ee08382081e3e060818ede497dbca26987dc
    > BUG: 1291259
    > Signed-off-by: Soumya Koduri <skoduri>
    > Reviewed-on: http://review.gluster.org/12962
    > CentOS-regression: Gluster Build System <jenkins.com>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > Smoke: Gluster Build System <jenkins.com>
    > Reviewed-by: Niels de Vos <ndevos>
    
    Change-Id: I7514ee08382081e3e060818ede497dbca26987dc
    BUG: 1347715
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/15600
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Kaushal M <kaushal>

--- Additional comment from Worker Ant on 2016-10-03 08:16:48 EDT ---

COMMIT: http://review.gluster.org/15601 committed in release-3.7 by Kaushal M (kaushal) 
------
commit 2cee4a3e47518aeb28ac2b611c6f01c0f9d00dab
Author: Niels de Vos <ndevos>
Date:   Sat Aug 6 16:04:29 2016 +0200

    libglusterfs: add gf_get_mem_type()
    
    gfapi needs to provide a function towards applications to free memory
    that it allocated. Depending on how the application is compiled/linked,
    it could use a different memory allocator than Gluster itself. Therefore
    it is not safe for gfapi to request applications to free memory with
    'standard' free().
    
    Examples for this are Gluster allocated structures with GF_CALLOC() when
    memory accounting is enabled (the default). Some gfapi functions use
    malloc() to allocate memory as a workaround, but the free() from the
    jemalloc implementation should not be combined with the malloc() from
    glibc.
    
    Cherry picked from commit db4e26ed71a01e5f760fbc3c7051962426f102c9:
    > Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
    > BUG: 1344714
    > Signed-off-by: Niels de Vos <ndevos>
    > Reviewed-on: http://review.gluster.org/15108
    > Smoke: Gluster Build System <jenkins.org>
    > Reviewed-by: soumya k <skoduri>
    > Reviewed-by: jiffin tony Thottan <jthottan>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Shyamsundar Ranganathan <srangana>
    
    Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
    BUG: 1347715
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/15601
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Kaushal M <kaushal>

--- Additional comment from Worker Ant on 2016-10-03 08:17:24 EDT ---

REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface for upcall consumers) posted (#3) for review on release-3.7 by Kaushal M (kaushal)

--- Additional comment from Worker Ant on 2016-10-03 08:20:47 EDT ---

COMMIT: http://review.gluster.org/15602 committed in release-3.7 by Kaushal M (kaushal) 
------
commit 55c92db32ba7d88359f0562953a3a6d8874dd1a5
Author: Niels de Vos <ndevos>
Date:   Fri Jun 10 18:23:43 2016 +0530

    gfapi: redesign the public interface for upcall consumers
    
    The glfs_callback_arg and glfs_callback_inode_arg were allocated by
    gfapi, and expected to be free()'d by the application. However it is not
    reasonable to expect that applications use the same memory allocator to
    as the compiled libgfapi.so. For instance, it is possible that gfapi
    uses glibc malloc/free, and an application like NFS-Ganesha the versions
    from jemalloc. Mismatching of the malloc() and free() functions causes
    segmentation faults at best.
    
    In order to prevent problems like this in the future, the API for
    applications that consume upcalls has been remodeled. Any of the
    structures that gfapi allocates, should be free'd with glfs_free(). The
    members of the structures can not be accessed directly anymore, each
    has its own function to access now.
    
    Correcting the naming of the functions, structures and constants is a
    continuation of commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6. These
    new improvements not only have correct prefixes for the functions and
    structures, the naming also reflects more to the upcall framework and
    does not use "callback" anymore.
    
    Cherry picked from commit 4721188a154acd9a0a4c096d8d73e97f3bf1b2a9:
    > Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
    > BUG: 1344714
    > Signed-off-by: Niels de Vos <ndevos>
    > Reviewed-on: http://review.gluster.org/14701
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Kaleb KEITHLEY <kkeithle>
    > Reviewed-by: soumya k <skoduri>
    > Reviewed-by: jiffin tony Thottan <jthottan>
    
    Once difference with the version of this change in other branches is
    that leases are not included in glusterfs-3.7. Hence there is a little
    change that drops the handling of GF_UPCALL_RECALL_LEASE.
    
    In addition, this backport contains commit 2775dc6410:
    > libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg
    > Reviewed-on: http://review.gluster.org/14702
    
    Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
    BUG: 1347715
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/15602
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Kaushal M <kaushal>

--- Additional comment from Kaushal on 2016-10-17 05:04:48 EDT ---

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.16, please open a new bug report.

glusterfs-3.7.16 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-devel/2016-October/051187.html
[2] https://www.gluster.org/pipermail/gluster-users/