Bug 1431787

Summary: calamari-lite: supervisord memory limit is too low
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Alexandre Marangone <amarango>
Component: CalamariAssignee: Boris Ranto <branto>
Calamari sub component: Back-end QA Contact: Vidushi Mishra <vimishra>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: branto, ceph-eng-bugs, gmeno, hnallurv, kdreyer
Version: 2.1   
Target Milestone: rc   
Target Release: 2.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: calamari-server-1.5.6-1.el7cp Ubuntu: calamari_1.5.6-2redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-19 13:30:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexandre Marangone 2017-03-13 18:19:59 UTC
Description of problem:
supervisord limits calamari-lite memory usage to 300M in /etc/systemd/system/supervisord.service.d/50-MemoryLimit.conf

On a cluster with 960 OSDs this created issues where requests to the calamari API would get stuck when importing the cluster to the console.
Changing the value to 1GB (guestimate) solved the issue.

Comment 2 Alexandre Marangone 2017-03-13 21:10:54 UTC
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1353987#c17

1GB is not enough.

Comment 3 Christina Meno 2017-04-25 16:35:03 UTC
It seems to me that just bumping up the memory limit is a bad idea considering that we are collocated with the monitor. 

Boris, would you please see what we can do to replicate this and reduce the memory consumed?

Comment 4 Boris Ranto 2017-04-25 16:43:20 UTC
We no longer use supervisord (we switched to systemd) and we do not enforce any memory limit in systemd so in a way, the issue is solved in 2.2.

I am wondering whether we do keep way too many references in memory, though.

@Alexandre: Does the memory gets freed after the import? Please note that it may take up to a minute or so for it to get freed thanks to the nature of garbage collection.

Comment 5 Boris Ranto 2017-04-25 23:14:50 UTC
It turns out we do enforce a memory limit, not in the service file but with calamari-ctl, upstream PR to change the limit to half the available memory:

https://github.com/ceph/calamari/pull/517

Comment 13 errata-xmlrpc 2017-06-19 13:30:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497