Bug 1431787 - calamari-lite: supervisord memory limit is too low
Summary: calamari-lite: supervisord memory limit is too low
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Calamari
Version: 2.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 2.3
Assignee: Boris Ranto
QA Contact: Vidushi Mishra
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-13 18:19 UTC by Alexandre Marangone
Modified: 2022-02-21 18:03 UTC (History)
5 users (show)

Fixed In Version: RHEL: calamari-server-1.5.6-1.el7cp Ubuntu: calamari_1.5.6-2redhat1xenial
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-19 13:30:29 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1497 0 normal SHIPPED_LIVE Red Hat Ceph Storage 2.3 bug fix and enhancement update 2017-06-19 17:24:11 UTC

Description Alexandre Marangone 2017-03-13 18:19:59 UTC
Description of problem:
supervisord limits calamari-lite memory usage to 300M in /etc/systemd/system/supervisord.service.d/50-MemoryLimit.conf

On a cluster with 960 OSDs this created issues where requests to the calamari API would get stuck when importing the cluster to the console.
Changing the value to 1GB (guestimate) solved the issue.

Comment 2 Alexandre Marangone 2017-03-13 21:10:54 UTC
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1353987#c17

1GB is not enough.

Comment 3 Christina Meno 2017-04-25 16:35:03 UTC
It seems to me that just bumping up the memory limit is a bad idea considering that we are collocated with the monitor. 

Boris, would you please see what we can do to replicate this and reduce the memory consumed?

Comment 4 Boris Ranto 2017-04-25 16:43:20 UTC
We no longer use supervisord (we switched to systemd) and we do not enforce any memory limit in systemd so in a way, the issue is solved in 2.2.

I am wondering whether we do keep way too many references in memory, though.

@Alexandre: Does the memory gets freed after the import? Please note that it may take up to a minute or so for it to get freed thanks to the nature of garbage collection.

Comment 5 Boris Ranto 2017-04-25 23:14:50 UTC
It turns out we do enforce a memory limit, not in the service file but with calamari-ctl, upstream PR to change the limit to half the available memory:

https://github.com/ceph/calamari/pull/517

Comment 13 errata-xmlrpc 2017-06-19 13:30:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497


Note You need to log in before you can comment on or make changes to this bug.