Description of problem:
supervisord limits calamari-lite memory usage to 300M in /etc/systemd/system/supervisord.service.d/50-MemoryLimit.conf
On a cluster with 960 OSDs this created issues where requests to the calamari API would get stuck when importing the cluster to the console.
Changing the value to 1GB (guestimate) solved the issue.
1GB is not enough.
It seems to me that just bumping up the memory limit is a bad idea considering that we are collocated with the monitor.
Boris, would you please see what we can do to replicate this and reduce the memory consumed?
We no longer use supervisord (we switched to systemd) and we do not enforce any memory limit in systemd so in a way, the issue is solved in 2.2.
I am wondering whether we do keep way too many references in memory, though.
@Alexandre: Does the memory gets freed after the import? Please note that it may take up to a minute or so for it to get freed thanks to the nature of garbage collection.
It turns out we do enforce a memory limit, not in the service file but with calamari-ctl, upstream PR to change the limit to half the available memory:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.