Bug 1368841 - Applications not calling glfs_h_poll_upcall() have upcall events cached for no use
Summary: Applications not calling glfs_h_poll_upcall() have upcall events cached for n...
Alias: None
Product: GlusterFS
Classification: Community
Component: libgfapi
Version: 3.8
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Niels de Vos
QA Contact: Sudhir D
Depends On: 1368842
TreeView+ depends on / blocked
Reported: 2016-08-21 19:37 UTC by Niels de Vos
Modified: 2016-09-16 18:27 UTC (History)
2 users (show)

Fixed In Version: glusterfs-3.8.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1368842 (view as bug list)
Last Closed: 2016-09-16 18:27:59 UTC
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:

Attachments (Terms of Use)

Description Niels de Vos 2016-08-21 19:37:10 UTC
Description of problem:
When a volume has upcalls (features.cache-invalidation) enabled, but does not call glfs_h_poll_upcall(), the upcall events accumulate in a list that is toed to the 'struct glfs'. The application might not be interested in the events, causing the list to grow indefinitely. This manifests in some sort of memory leak.

Version-Release number of selected component (if applicable):
3.7 and upwards

How reproducible:

Steps to Reproduce:
1. install glusterfs-coreutils and open a connection like
      # gfcli glfs://localhost/test-volume
2. mount the volume over fuse
3. over fuse, create a file like
      # while sleep 0.01 ; do date >> TIME ; done
4. over the glfcli, do a regular 'tail TIME'
5. in a 3rd terminal, watch 'top -p $(pidof gfcli)' increasing memory

Actual results:
Memory of gfcli grows slowly, but constantly.

Expected results:
gfcli should have stable memory usage.

Additional info:
Reported in relation to md-cache improvements by Poornima.

Comment 1 Worker Ant 2016-08-30 08:36:23 UTC
REVIEW: http://review.gluster.org/15346 (gfapi: do not cache upcalls if the application is not interested) posted (#1) for review on release-3.8 by Niels de Vos (ndevos@redhat.com)

Comment 2 Worker Ant 2016-08-31 07:38:20 UTC
COMMIT: http://review.gluster.org/15346 committed in release-3.8 by Niels de Vos (ndevos@redhat.com) 
commit 3eecd4e2b5a923d7a5da1b899b6f0009e319b98d
Author: Niels de Vos <ndevos@redhat.com>
Date:   Tue Aug 30 10:23:55 2016 +0200

    gfapi: do not cache upcalls if the application is not interested
    When the volume option 'features.cache-invalidation' is enabled, upcall
    events are sent from the brick process to the client. Even if the client
    is not interested in upcall events itself, md-cache or other xlators may
    benefit from them.
    By adding a new 'cache_upcalls' boolean in the 'struct glfs', we can
    enable the caching of upcalls when the application called
    glfs_h_poll_upcall(). NFS-Ganesha sets up a thread for handling upcalls
    in the initialization phase, and calls glfs_h_poll_upcall() before any
    NFS-client accesses the NFS-export.
    In the future there will be a more flexible registration API for
    enabling certain kind of upcall events. Until that is available, this
    should work just fine.
    Verificatio of this change is not trivial within our current regression
    test framework. The bug report contains a description on how to reliably
    reproduce the problem with the glusterfs-coreutils.
    Cherry picked from commit 218c9b033fa44eacbc27d87491abd830548b362e:
    > Change-Id: I818595c92db50e6e48f7bfe287ee05103a4a30a2
    > BUG: 1368842
    > Signed-off-by: Niels de Vos <ndevos@redhat.com>
    > Reviewed-on: http://review.gluster.org/15191
    > Smoke: Gluster Build System <jenkins@build.gluster.org>
    > Reviewed-by: Poornima G <pgurusid@redhat.com>
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    > Reviewed-by: soumya k <skoduri@redhat.com>
    > Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
    Change-Id: I818595c92db50e6e48f7bfe287ee05103a4a30a2
    BUG: 1368841
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: http://review.gluster.org/15346
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: soumya k <skoduri@redhat.com>

Comment 3 Niels de Vos 2016-09-12 05:39:35 UTC
All 3.8.x bugs are now reported against version 3.8 (without .x). For more information, see http://www.gluster.org/pipermail/gluster-devel/2016-September/050859.html

Comment 4 Niels de Vos 2016-09-16 18:27:59 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.4, please open a new bug report.

glusterfs-3.8.4 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/announce/2016-September/000060.html
[2] https://www.gluster.org/pipermail/gluster-users/

Note You need to log in before you can comment on or make changes to this bug.