Bug 1672258

Created attachment 1526739 [details]
statedump 20190204

Description of problem:
Fuse will take daily more and more memory until the swap is full. Then the system won't work properly anymore.

We have upgraded to 4.1 at the end of December 2018 and since there we have to restart the gluster cluster and all nodes more or less every 2 weeks, because the memory is taken. We have this situation on all div. gluster clusters.

We have gluster 4.1.7 on a ubuntu 16.04.5 LTS (xenial) installed.


System-Checks:

---
Memory and Swap:
free
              total        used        free      shared  buff/cache   available
Mem:       32834992    31387196      243932        9148     1203864      897800
Swap:      31999996    25951268     6048728

---
top and find out the service and get the status of the service

systemctl status data_net.mount
● data_net.mount - Mount System glusterfs on path /data_net from source localhost:/ctgv0  with
   Loaded: loaded (/etc/systemd/system/data_net.mount; static; vendor preset: enabled)
   Active: active (mounted) since Fri 2019-02-01 07:51:32 CET; 3 days ago
    Where: /data_net
     What: localhost:/ctgv0
     Docs: https://oguya.ch/posts/2015-09-01-systemd-mount-partition/
  Process: 11256 ExecUnmount=/bin/umount /data_net (code=exited, status=0/SUCCESS)
  Process: 11257 ExecMount=/bin/mount localhost:/ctgv0 /data_net -t glusterfs -o defaults,_netdev (code=
    Tasks: 20
   Memory: 28.0G
      CPU: 12h 43min 23.929s
   CGroup: /system.slice/data_net.mount
           ├─ 7825 /usr/sbin/glusterfs --process-name fuse --volfile-server=localhost --volfile-id=/ctgv
           └─11337 /usr/sbin/glusterfs --process-name fuse --volfile-server=localhost --volfile-id=/ctgv

Feb 01 07:51:32 nucprdstk112 systemd[1]: Mounting coretech: Mount System glusterfs on path /data_net fro
Feb 01 07:51:32 nucprdstk112 systemd[1]: Mounted coretech: Mount System glusterfs on path /data_net from
Feb 01 08:02:08 nucprdstk112 data_net[7825]: [2019-02-01 07:02:08.392799] C [rpc-clnt-ping.c:166:rpc_cln

---
uptime:
 12:03:44 up 17 days, 16 min,  1 user,  load average: 1.30, 1.03, 1.01

---
Followed this description:
https://docs.gluster.org/en/v3/Troubleshooting/troubleshooting-memory/

gluster volume info

Volume Name: ctgv0
Type: Replicate
Volume ID: 0e70a1ba-2c70-494a-8a85-f757fe77901a
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: nucprdstk112:/var/glusterfs/ctgv0/brick1
Brick2: nucprdstk113:/var/glusterfs/ctgv0/brick2
Brick3: nucprdstk114:/var/glusterfs/ctgv0/brick2
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet


Version-Release number of selected component (if applicable):

gluster --version
glusterfs 4.1.7
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial





How reproducible:


Steps to Reproduce:
1. Restart server and wait for 1 or 2 weeks
2.
3.

Actual results:
still takes memory every day.


Expected results:
free memory


Additional info: