Bug 1654642

Summary: Very high memory usage with glusterfs VFS module
Product: [Community] GlusterFS Reporter: ryan
Component: gluster-smbAssignee: bugs <bugs>
Status: CLOSED DUPLICATE QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.12CC: bugs, pasik
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-07 11:32:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
pmap output from Samba PID
none
'smbcontrol 98757 pool-usage' output
none
'strace -ttT -f -o /tmp/control.strace smbcontrol 98757 pool-usage' output
none
VFS write test results
none
Python file for replicating issue none

Description ryan 2018-11-29 10:19:58 UTC
Created attachment 1509774 [details]
pmap output from Samba PID

Description of problem:
Very high memory usage on all SMB PIDs

Version-Release number of selected component (if applicable):
Samba 4.8.6
Gluster 3.12.14

How reproducible:
Happens overtime, especially when a lot of write operations are occurring on the node via SMB

Steps to Reproduce:
1.Multiple clients connect to SMB share(s)
2.Copy multiple large files to shares
3.Check PID usage through htop or top
4.Analyse memory allocation via pmap

Actual results:
Over 36GB of memory consumed by one PID

Expected results:
Much lower memory footprint per PID

Comment 1 ryan 2018-12-13 15:38:21 UTC
After some analysis of the SMBD process, the Samba team have said that the core SMB process is only using around 3MB of memory. So something in the gluster VFS must be consuming this excess amount of memory.
Any help you could provide would be much appreciated. Please let me know if you need any data.

Best regards,
Ryan

Comment 2 ryan 2018-12-14 10:47:13 UTC
Created attachment 1514324 [details]
'smbcontrol 98757 pool-usage' output

Comment 3 ryan 2018-12-14 10:48:11 UTC
Created attachment 1514325 [details]
'strace -ttT -f -o /tmp/control.strace smbcontrol 98757 pool-usage' output

Comment 4 ryan 2018-12-17 11:04:29 UTC
After some internal testing, we've found this issue is easily repeatable over time.
Running the Python script (Attached) over the weekend, we got the SMBD thread up to 12GB of memory usage.

I've also uploaded out test results in PDF form.

Any advice would be much appreciated,
Best,
Ryan

Comment 5 ryan 2018-12-17 11:05:19 UTC
Created attachment 1515041 [details]
VFS write test results

Comment 6 ryan 2018-12-17 11:11:30 UTC
Created attachment 1515042 [details]
Python file for replicating issue

Just need to change line 6 to point to a VFS share.
This was tested on Windows Server 2016.

Test will run until manually stopped.

Comment 7 ryan 2019-01-07 11:32:20 UTC
Due to lack of responses and discoveries found during testing, I'm closing this ticket and have opened another (1663519)

*** This bug has been marked as a duplicate of bug 1663519 ***