Bug 1596746 - High Memory Utilization Glusterfs
Summary: High Memory Utilization Glusterfs
Alias: None
Product: GlusterFS
Classification: Community
Component: core
Version: mainline
Hardware: x86_64
OS: Linux
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2018-06-29 15:11 UTC by dscott
Modified: 2019-01-18 23:01 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-01-18 23:01:37 UTC
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:

Attachments (Terms of Use)
Screenshot of hostnamectl and glusterfs status (51.36 KB, image/png)
2018-06-29 15:11 UTC, dscott
no flags Details
Screenshot of glusterfs volume status (125.79 KB, image/png)
2018-06-29 15:13 UTC, dscott
no flags Details
Screenshot of top processes sorted by RES (245.97 KB, image/png)
2018-07-02 12:33 UTC, dscott
no flags Details

Description dscott 2018-06-29 15:11:39 UTC
Created attachment 1455528 [details]
Screenshot of hostnamectl and glusterfs status

Description of problem:
I have three glusterfs servers, all replicated just two bricks across all three servers.  All three servers have memory utilization consistently at 95% to 98%.  The servers are about 70% free space.

The gluster servers were deployed using Ovirt gui.   All three servers are experiencing high utilization.  I have been rebooting the servers weekly scared servers will crash.  

The servers do only have 64MB of RAM.   How can I determine if I should add more RAM.  I do not want to add more RAM and this issue not be resolved.

Version-Release number of selected component (if applicable):
Glsuterfs version 3.12.9
Centos 7.4 servers

How reproducible:

The problem is consistent

Comment 2 dscott 2018-06-29 15:13:16 UTC
Created attachment 1455529 [details]
Screenshot of glusterfs volume status

Comment 3 Sunil Kumar Acharya 2018-06-29 16:25:50 UTC
Closing this bug. Please raise an upstream bug for all the upstream gluster

Comment 4 Atin Mukherjee 2018-07-02 00:15:12 UTC
Moved this to upstream.

Comment 5 Atin Mukherjee 2018-07-02 02:42:38 UTC
Could you please cross check which process was consuming higher memory, is it glusterd or glusterfsd? Attaching the statedump of the same process would be helpful for engineers to debug this further.

Comment 6 dscott 2018-07-02 12:32:02 UTC
So, I have several glusterfsd processes which are using minimal memory.  I really don't have any processes which are consuming exorbitant amount of memory.  My servers all show usage in the 95%.  I have attached a top listing sorted by resident memory.

Comment 7 dscott 2018-07-02 12:33:26 UTC
Created attachment 1455950 [details]
Screenshot of top processes sorted by RES

Comment 8 Amar Tumballi 2018-09-18 13:45:23 UTC
3.12.9 seems to have gotten a leak bug introduced. Please consider upgrading to 3.12.13/3.12.14 for getting that leak fixed.

Comment 9 Shyamsundar 2018-10-23 14:54:30 UTC
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.

Comment 10 dscott 2019-01-18 23:01:37 UTC
I updated to latest gluster and no longer having memory high memory issues.

Note You need to log in before you can comment on or make changes to this bug.