Created attachment 1453582 [details] Gluster dump of client fuse process Description of problem: Gluster-fuse mount process is consuming large amounts of memory over a relatively short period of time (consuming GBs over a day) on a mount < 100MB but with lots of churn. Version-Release number of selected component (if applicable): # Client side glusterfs-3.12.10-1.el7.x86_64 glusterfs-client-xlators-3.12.10-1.el7.x86_64 glusterfs-libs-3.12.10-1.el7.x86_64 glusterfs-fuse-3.12.10-1.el7.x86_64 # Server Side: glusterfs-cli-3.12.10-1.el7.x86_64 glusterfs-3.12.10-1.el7.x86_64 glusterfs-fuse-3.12.10-1.el7.x86_64 glusterfs-libs-3.12.10-1.el7.x86_64 glusterfs-api-3.12.10-1.el7.x86_64 glusterfs-client-xlators-3.12.10-1.el7.x86_64 glusterfs-server-3.12.10-1.el7.x86_64 # Volume setup: # 3 node 3 brick replica, has KahaDB files for activeMQ on it, mount itself is only a few MB used: node-001:/gv_activemq 250G 84M 250G 1% /mnt/amq_broker # vol info: Volume Name: gv_activemq Type: Replicate Volume ID: d3003a1d-b07e-4996-998d-6cbe26a587e2 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: node-001:/opt/gluster_storage/gv_activemq/brick Brick2: node-002:/opt/gluster_storage/gv_activemq/brick Brick3: node-003:/opt/gluster_storage/gv_activemq/brick Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet performance.readdir-ahead: off network.ping-timeout: 5 performance.cache-size: 1GB How reproducible: Memory utilisation seems to be a problem in at least 3.12.9 and 10 (I've upgraded from .9 to .10 in hopes that this would fix it). I've got another cluster running 3.12.6 whose client doesn't have the same issue and is running the other end of this ActiveMQ cluster). So looks to have been introduced since then. Steps to Reproduce: 1. ? 2. 3. Expected results: Additional info: # 3.12.6-1 usage on a much busier mount: 43489 root 20 0 681796 53828 4328 S 0.3 0.3 1624:11 glusterfs node-001:/gv_amq_broker 200G 36G 165G 18% /opt/amq_broker # 3.12.9/10 on a similar mount with less traffic: 48376 root 20 0 5164844 4.038g 4460 S 5.4 26.0 32:11.30 glusterfs node-001:/gv_activemq 250G 84M 250G 1% /mnt/amq_broker
just a comment, state dump provided was after the upgrade from 3.12.9 (the 4G res memory above) the memory footprint for the 3.12.10 version was: 8499 root 20 0 2123592 1.436g 4124 S 0.0 9.3 18:56.34 glusterfs when the statedump was taken.
I have the same issue with steadily increasing client memory usage on 3.12.9 under read/write load. I can add that I'm using GlusterFS as oVirt VM storage, so I tried switching oVirt to using libgfapi instead of using the fuse mounts, but that just moved the memory leak to the individual qemu processes instead of being centralised to the fuse mount processes. That also confirmed that it seems to be mostly database servers that cause the issue (so the oVirt hosted engine, and some other PostgreSQL servers I'm running that see a lot of traffic). I'm currently working around the issue by migrating VMs manually between servers in the cluster, but it's annoying to have to do that twice a day to keep the servers from using 100% memory and VMs getting killed. I'm surprised there hasn't been more discussion on this issue, is it some setting that we have that's causing this? The engine gluster volume configuration: Volume Name: engine Type: Replicate Volume ID: 84f29251-619b-493c-ae1c-7da0fd27a8c1 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 5 = 5 Transport-type: tcp Bricks: Brick1: gluster1.management:/gluster/sda/engine Brick2: gluster2.management:/gluster/sda/engine Brick3: gluster3.management:/gluster/sda/engine Brick4: gluster4.management:/gluster/sda/engine Brick5: gluster5.management:/gluster/sdb/engine Options Reconfigured: server.allow-insecure: on performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server network.ping-timeout: 10 storage.owner-uid: 36 storage.owner-gid: 36 performance.flush-behind: on performance.low-prio-threads: 32 cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off nfs.disable: on transport.address-family: inet cluster.server-quorum-ratio: 51%
We've migrated the affected cluster to the new 4.1 release and it seems to have "fixed" the issue. the 3.9/10 releases were unusable for us as we risked OOMing if we left the mount running too long.
Hi, Thanks for the update. Helps us to corner the specific patch, so hopefully we can fix 3.12.x branch.
After downgrading to 3.12.6, it seems the issue is no longer present. Downgrading to 3.12.8 did not help. It doesn't look like 3.12.7 got to the CentOS repos, so I haven't tested that version.
I'm running oVirt cluster with 3 nodes and after around a week of use, the hosted_storage gluster monut consumes around 16GB of memory on each host.
The same is happening on Debian 9 with GlusterFS 3.12.12 - RSS memory of the glusterfs process (fuse client) is gradually rising until the server runs out of memory. I tried to downgrade both glusterfs-server on the servers and glusterfs-client on the client to 3.12.11, then 3.12.6, 3.12.5, 3.12.4, 3.12.3, none of these worked. Currently Glusterfs 3.12.x is unusable on Debian 9.
I've opened a backport request here as it seems a commit has been made to fix this, but isn't in the current 3.12.x release: https://bugzilla.redhat.com/show_bug.cgi?id=1613512
This is fixed with 3.12.13 now.
*** Bug 1583502 has been marked as a duplicate of this bug. ***
*** Bug 1589090 has been marked as a duplicate of this bug. ***