Bug 1596746

Summary: High Memory Utilization Glusterfs
Product: [Community] GlusterFS Reporter: dscott
Component: coreAssignee: bugs <bugs>
Status: CLOSED NOTABUG QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: mainlineCC: atumball, bugs, dscott, pasik, rhs-bugs, sankarshan, sheggodu, storage-qa-internal, vbellur
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-18 23:01:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Screenshot of hostnamectl and glusterfs status
none
Screenshot of glusterfs volume status
none
Screenshot of top processes sorted by RES none

Description dscott 2018-06-29 15:11:39 UTC
Created attachment 1455528 [details]
Screenshot of hostnamectl and glusterfs status

Description of problem:
I have three glusterfs servers, all replicated just two bricks across all three servers.  All three servers have memory utilization consistently at 95% to 98%.  The servers are about 70% free space.

The gluster servers were deployed using Ovirt gui.   All three servers are experiencing high utilization.  I have been rebooting the servers weekly scared servers will crash.  

The servers do only have 64MB of RAM.   How can I determine if I should add more RAM.  I do not want to add more RAM and this issue not be resolved.


Version-Release number of selected component (if applicable):
Glsuterfs version 3.12.9
Centos 7.4 servers



How reproducible:

The problem is consistent

Comment 2 dscott 2018-06-29 15:13:16 UTC
Created attachment 1455529 [details]
Screenshot of glusterfs volume status

Comment 3 Sunil Kumar Acharya 2018-06-29 16:25:50 UTC
Closing this bug. Please raise an upstream bug for all the upstream gluster
versions.

Comment 4 Atin Mukherjee 2018-07-02 00:15:12 UTC
Moved this to upstream.

Comment 5 Atin Mukherjee 2018-07-02 02:42:38 UTC
Could you please cross check which process was consuming higher memory, is it glusterd or glusterfsd? Attaching the statedump of the same process would be helpful for engineers to debug this further.

Comment 6 dscott 2018-07-02 12:32:02 UTC
So, I have several glusterfsd processes which are using minimal memory.  I really don't have any processes which are consuming exorbitant amount of memory.  My servers all show usage in the 95%.  I have attached a top listing sorted by resident memory.

Comment 7 dscott 2018-07-02 12:33:26 UTC
Created attachment 1455950 [details]
Screenshot of top processes sorted by RES

Comment 8 Amar Tumballi 2018-09-18 13:45:23 UTC
3.12.9 seems to have gotten a leak bug introduced. Please consider upgrading to 3.12.13/3.12.14 for getting that leak fixed.

Comment 9 Shyamsundar 2018-10-23 14:54:30 UTC
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.

Comment 10 dscott 2019-01-18 23:01:37 UTC
I updated to latest gluster and no longer having memory high memory issues.