Bug 1654642 - Very high memory usage with glusterfs VFS module
Summary: Very high memory usage with glusterfs VFS module
Keywords:
Status: CLOSED DUPLICATE of bug 1663519
Alias: None
Product: GlusterFS
Classification: Community
Component: gluster-smb
Version: 3.12
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-29 10:19 UTC by ryan
Modified: 2019-01-07 11:32 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2019-01-07 11:32:20 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)
pmap output from Samba PID (65.40 KB, text/plain)
2018-11-29 10:19 UTC, ryan
no flags Details
'smbcontrol 98757 pool-usage' output (782.76 KB, text/plain)
2018-12-14 10:47 UTC, ryan
no flags Details
'strace -ttT -f -o /tmp/control.strace smbcontrol 98757 pool-usage' output (367.74 KB, text/plain)
2018-12-14 10:48 UTC, ryan
no flags Details
VFS write test results (62.11 KB, application/pdf)
2018-12-17 11:05 UTC, ryan
no flags Details
Python file for replicating issue (547 bytes, text/plain)
2018-12-17 11:11 UTC, ryan
no flags Details

Description ryan 2018-11-29 10:19:58 UTC
Created attachment 1509774 [details]
pmap output from Samba PID

Description of problem:
Very high memory usage on all SMB PIDs

Version-Release number of selected component (if applicable):
Samba 4.8.6
Gluster 3.12.14

How reproducible:
Happens overtime, especially when a lot of write operations are occurring on the node via SMB

Steps to Reproduce:
1.Multiple clients connect to SMB share(s)
2.Copy multiple large files to shares
3.Check PID usage through htop or top
4.Analyse memory allocation via pmap

Actual results:
Over 36GB of memory consumed by one PID

Expected results:
Much lower memory footprint per PID

Comment 1 ryan 2018-12-13 15:38:21 UTC
After some analysis of the SMBD process, the Samba team have said that the core SMB process is only using around 3MB of memory. So something in the gluster VFS must be consuming this excess amount of memory.
Any help you could provide would be much appreciated. Please let me know if you need any data.

Best regards,
Ryan

Comment 2 ryan 2018-12-14 10:47:13 UTC
Created attachment 1514324 [details]
'smbcontrol 98757 pool-usage' output

Comment 3 ryan 2018-12-14 10:48:11 UTC
Created attachment 1514325 [details]
'strace -ttT -f -o /tmp/control.strace smbcontrol 98757 pool-usage' output

Comment 4 ryan 2018-12-17 11:04:29 UTC
After some internal testing, we've found this issue is easily repeatable over time.
Running the Python script (Attached) over the weekend, we got the SMBD thread up to 12GB of memory usage.

I've also uploaded out test results in PDF form.

Any advice would be much appreciated,
Best,
Ryan

Comment 5 ryan 2018-12-17 11:05:19 UTC
Created attachment 1515041 [details]
VFS write test results

Comment 6 ryan 2018-12-17 11:11:30 UTC
Created attachment 1515042 [details]
Python file for replicating issue

Just need to change line 6 to point to a VFS share.
This was tested on Windows Server 2016.

Test will run until manually stopped.

Comment 7 ryan 2019-01-07 11:32:20 UTC
Due to lack of responses and discoveries found during testing, I'm closing this ticket and have opened another (1663519)

*** This bug has been marked as a duplicate of bug 1663519 ***


Note You need to log in before you can comment on or make changes to this bug.