Created attachment 1526739 [details] statedump 20190204 Description of problem: Fuse will take daily more and more memory until the swap is full. Then the system won't work properly anymore. We have upgraded to 4.1 at the end of December 2018 and since there we have to restart the gluster cluster and all nodes more or less every 2 weeks, because the memory is taken. We have this situation on all div. gluster clusters. We have gluster 4.1.7 on a ubuntu 16.04.5 LTS (xenial) installed. System-Checks: --- Memory and Swap: free total used free shared buff/cache available Mem: 32834992 31387196 243932 9148 1203864 897800 Swap: 31999996 25951268 6048728 --- top and find out the service and get the status of the service systemctl status data_net.mount ● data_net.mount - Mount System glusterfs on path /data_net from source localhost:/ctgv0 with Loaded: loaded (/etc/systemd/system/data_net.mount; static; vendor preset: enabled) Active: active (mounted) since Fri 2019-02-01 07:51:32 CET; 3 days ago Where: /data_net What: localhost:/ctgv0 Docs: https://oguya.ch/posts/2015-09-01-systemd-mount-partition/ Process: 11256 ExecUnmount=/bin/umount /data_net (code=exited, status=0/SUCCESS) Process: 11257 ExecMount=/bin/mount localhost:/ctgv0 /data_net -t glusterfs -o defaults,_netdev (code= Tasks: 20 Memory: 28.0G CPU: 12h 43min 23.929s CGroup: /system.slice/data_net.mount ├─ 7825 /usr/sbin/glusterfs --process-name fuse --volfile-server=localhost --volfile-id=/ctgv └─11337 /usr/sbin/glusterfs --process-name fuse --volfile-server=localhost --volfile-id=/ctgv Feb 01 07:51:32 nucprdstk112 systemd[1]: Mounting coretech: Mount System glusterfs on path /data_net fro Feb 01 07:51:32 nucprdstk112 systemd[1]: Mounted coretech: Mount System glusterfs on path /data_net from Feb 01 08:02:08 nucprdstk112 data_net[7825]: [2019-02-01 07:02:08.392799] C [rpc-clnt-ping.c:166:rpc_cln --- uptime: 12:03:44 up 17 days, 16 min, 1 user, load average: 1.30, 1.03, 1.01 --- Followed this description: https://docs.gluster.org/en/v3/Troubleshooting/troubleshooting-memory/ gluster volume info Volume Name: ctgv0 Type: Replicate Volume ID: 0e70a1ba-2c70-494a-8a85-f757fe77901a Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: nucprdstk112:/var/glusterfs/ctgv0/brick1 Brick2: nucprdstk113:/var/glusterfs/ctgv0/brick2 Brick3: nucprdstk114:/var/glusterfs/ctgv0/brick2 Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet Version-Release number of selected component (if applicable): gluster --version glusterfs 4.1.7 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.5 LTS Release: 16.04 Codename: xenial How reproducible: Steps to Reproduce: 1. Restart server and wait for 1 or 2 weeks 2. 3. Actual results: still takes memory every day. Expected results: free memory Additional info:
Created attachment 1526740 [details] statedump 20190201 another statedump file from 1st February Thanks a lot for your advice / support Ritzo
With patch https://review.gluster.org/#/q/Ifee0737b23b12b1426c224ec5b8f591f487d83a2 merged in glusterfs-6.0 and glusterfs-5.5, this should be now fixed. Please upgrade and test it for us.
Closing the issue as CURRENTRELEASE with above comment's data. Please upgrade.