Bug 1378948 - removal of file from nfs mount crashes ganesha server
Summary: removal of file from nfs mount crashes ganesha server
Keywords:
Status: CLOSED EOL
Alias: None
Product: GlusterFS
Classification: Community
Component: upcall
Version: 3.9
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Niels de Vos
QA Contact:
URL:
Whiteboard:
Depends On: 1344714
Blocks: glusterfs-3.9.0 1397506
TreeView+ depends on / blocked
 
Reported: 2016-09-23 15:23 UTC by Niels de Vos
Modified: 2017-03-08 12:34 UTC (History)
2 users (show)

Fixed In Version:
Clone Of: 1344714
Environment:
Last Closed: 2017-03-08 12:34:35 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Niels de Vos 2016-09-23 15:23:47 UTC
+++ This bug was initially created as a clone of Bug #1344714 +++

Description of problem:

The struct callback_inode_arg is a variable used by upcall infra for populating upcall entries. This variable is allocated using GF_CALLOC in glfs_h_poll_cache_invalidation(), but it also freedin FSAL_GLUSTER using free.
This will result in a crash when we try to remove a file from mount

Version-Release number of selected component (if applicable):
mainline

How reproducible:


Steps to Reproduce:
1. enable nfs-ganesha and export a volume 
2. mount the volume
3. create a file and delete it from the mount

Actual results:
ganesha server crashes

Expected results:
should not crash

Additional info:

--- Additional comment from Vijay Bellur on 2016-06-10 23:09:37 CEST ---

COMMIT: http://review.gluster.org/14702 committed in master by Jeff Darcy (jdarcy) 
------
commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6
Author: Jiffin Tony Thottan <jthottan>
Date:   Fri Jun 10 18:23:43 2016 +0530

    libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg
    
    Change-Id: I371525775db4f6a4d69beb94baaa53d17b16fb41
    BUG: 1344714
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14702
    CentOS-regression: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jdarcy>
    Tested-by: Jeff Darcy <jdarcy>
    Smoke: Gluster Build System <jenkins.com>

--- Additional comment from Vijay Bellur on 2016-08-15 09:56:58 CEST ---

COMMIT: http://review.gluster.org/15108 committed in master by Niels de Vos (ndevos) 
------
commit db4e26ed71a01e5f760fbc3c7051962426f102c9
Author: Niels de Vos <ndevos>
Date:   Sat Aug 6 16:04:29 2016 +0200

    libglusterfs: add gf_get_mem_type()
    
    gfapi needs to provide a function towards applications to free memory
    that it allocated. Depending on how the application is compiled/linked,
    it could use a different memory allocator than Gluster itself. Therefore
    it is not safe for gfapi to request applications to free memory with
    'standard' free().
    
    Examples for this are Gluster allocated structures with GF_CALLOC() when
    memory accounting is enabled (the default). Some gfapi functions use
    malloc() to allocate memory as a workaround, but the free() from the
    jemalloc implementation should not be combined with the malloc() from
    glibc.
    
    Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
    BUG: 1344714
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/15108
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: soumya k <skoduri>
    Reviewed-by: jiffin tony Thottan <jthottan>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana>

--- Additional comment from Worker Ant on 2016-09-18 10:16:09 CEST ---

REVIEW: http://review.gluster.org/14701 (gfapi: redesign the public interface for upcall consumers) posted (#15) for review on master by Niels de Vos (ndevos)

Comment 1 Worker Ant 2016-09-29 08:23:32 UTC
REVIEW: http://review.gluster.org/15597 (gfapi: redesign the public interface for upcall consumers) posted (#1) for review on release-3.9 by Niels de Vos (ndevos)

Comment 2 Worker Ant 2016-09-30 10:51:55 UTC
REVIEW: http://review.gluster.org/15597 (gfapi: redesign the public interface for upcall consumers) posted (#2) for review on release-3.9 by Niels de Vos (ndevos)

Comment 3 Worker Ant 2016-10-11 12:19:00 UTC
COMMIT: http://review.gluster.org/15597 committed in release-3.9 by Kaleb KEITHLEY (kkeithle) 
------
commit 11279aacbda1f05c3405a85583973f24ce20d103
Author: Niels de Vos <ndevos>
Date:   Thu Sep 29 10:19:26 2016 +0200

    gfapi: redesign the public interface for upcall consumers
    
    The glfs_callback_arg and glfs_callback_inode_arg were allocated by
    gfapi, and expected to be free()'d by the application. However it is not
    reasonable to expect that applications use the same memory allocator to
    as the compiled libgfapi.so. For instance, it is possible that gfapi
    uses glibc malloc/free, and an application like NFS-Ganesha the versions
    from jemalloc. Mismatching of the malloc() and free() functions causes
    segmentation faults at best.
    
    In order to prevent problems like this in the future, the API for
    applications that consume upcalls has been remodeled. Any of the
    structures that gfapi allocates, should be free'd with glfs_free(). The
    members of the structures can not be accessed directly anymore, each
    has its own function to access now.
    
    Correcting the naming of the functions, structures and constants is a
    continuation of commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6. These
    new improvements not only have correct prefixes for the functions and
    structures, the naming also reflects more to the upcall framework and
    does not use "callback" anymore.
    
    Cherry picked from commit 4721188a154acd9a0a4c096d8d73e97f3bf1b2a9:
    > Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
    > BUG: 1344714
    > Signed-off-by: Niels de Vos <ndevos>
    > Reviewed-on: http://review.gluster.org/14701
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Kaleb KEITHLEY <kkeithle>
    > Reviewed-by: soumya k <skoduri>
    > Reviewed-by: jiffin tony Thottan <jthottan>
    
    Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
    BUG: 1378948
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/15597
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: jiffin tony Thottan <jthottan>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>

Comment 4 Aravinda VK 2016-10-27 05:22:41 UTC
glusterfs-3.9.0rc2 is released[1] and packages are available for different distributions[2] to test.

[1] http://www.gluster.org/pipermail/maintainers/2016-October/001601.html
[2] http://www.gluster.org/pipermail/maintainers/2016-October/001605.html and http://www.gluster.org/pipermail/maintainers/2016-October/001606.html

Comment 5 Worker Ant 2016-11-04 07:26:38 UTC
REVIEW: http://review.gluster.org/15774 (gfapi/upcall: Fix mismatch in few upcall API SYMVER) posted (#1) for review on release-3.9 by soumya k (skoduri)

Comment 6 Worker Ant 2016-11-06 13:24:40 UTC
COMMIT: http://review.gluster.org/15774 committed in release-3.9 by Kaleb KEITHLEY (kkeithle) 
------
commit d569483536da1e8017f7105de8b211f172438d4c
Author: Soumya Koduri <skoduri>
Date:   Tue Nov 1 20:32:17 2016 +0530

    gfapi/upcall: Fix mismatch in few upcall API SYMVER
    
    There is mismatch in few of the upcall API routine
    definitions and their corresponding symbol version
    declarations. Fixed the same.
    
    > Change-Id: I2edfd9546a4c6a9128757f3b68e3ae4edd2c7a79
    > BUG: 1344714
    > Signed-off-by: Soumya Koduri <skoduri>
    > Reviewed-on: http://review.gluster.org/15760
    > Smoke: Gluster Build System <jenkins.org>
    > Reviewed-by: jiffin tony Thottan <jthottan>
    > Reviewed-by: Kaleb KEITHLEY <kkeithle>
    > Tested-by: Kaleb KEITHLEY <kkeithle>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > (cherry picked from commit 5b453380b15f34c0841f453a02fdabc46ad2f7ef)
    
    Change-Id: Ia9f5ab5d8d7d4ad77f1215e8249d346e31080573
    BUG: 1378948
    Signed-off-by: Soumya Koduri <skoduri>
    Reviewed-on: http://review.gluster.org/15774
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>

Comment 7 Kaushal 2017-03-08 12:34:35 UTC
This bug is getting closed because GlusterFS-3.9 has reached its end-of-life [1].

Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS.
If this bug still exists in newer GlusterFS releases, please open a new bug against the newer release.

[1]: https://www.gluster.org/community/release-schedule/


Note You need to log in before you can comment on or make changes to this bug.