Description of problem:
Customer has the native client running inside of a VM. The process consumes 4GB of memory very stable over a 6 month period.
Since the consumed memory is not marked as cache, the 'free' command shows this memory as 'used' and monitoring is alarming the customer of filled up memory when it reaches the 80% mark.
We need a way to display the currently allocated cache size of the client in a simple CLI command, so we can subtract this from the used memory count.
As a workaround it would be also possible to limit the amount of memory available for the client in total (memory per file is not usable)
Version-Release number of selected component (if applicable):
Statedump of native client process will be uploaded by the customer directly at a later point in time.
Created attachment 1168239 [details]
State dump RHGS1 from CLIENT15
Created attachment 1168240 [details]
State dump RHGS2 from CLIENT15
I've attached both state dumps from the two RHGS server.
You can see here below the memory usage of the client.
The mountpoint is used by an FTP server and a Apache to serve static content. We're talking about small files here.
CLIENT15# ps auxf | grep gluster
root 15015 0.0 0.0 103308 832 pts/0 S+ 09:33 0:00 \_ grep gluster
root 39521 1.2 50.9 5605112 4105776 ? Ssl Apr28 853:51 /usr/sbin/glusterfs --selinux --volfile-server=RHGS1 --volfile-server=RHGS2 --volfile-id=VOLUME1 /MOUNTPOINT
CLIENT15# free -m
total used free shared buffers cached
Mem: 7864 7725 139 0 127 3112
-/+ buffers/cache: 4485 3379
Swap: 4095 44 4051
kill -SIGUSR1 $(pgrep glusterfs)
CLIENT15# kill -SIGUSR1 $(pgrep glusterfs)
The state dumps which you provided are from brick process (please see comment8 and comment11). The comments you provided to take statedump is correct, you might have uploaded a wrong one.
If you can upload a statedump for client process when the memory usage is high, that would really help to find the root cause.
I did not find any other dump anywhere. No dump was find on the client (including /tmp and /var/run/).
How should it be done and find then ?
For the record, the memory consumption is always high. It suffice to access files on the filesystem to have it growing.
Upstream design discussion thread https://firstname.lastname@example.org/msg25851.html.
If you have any suggestion , feel free to reply to thread.