Bug 1505452 - Start ovirt-engine and ovirt-engine-notifier services by default with MALLOC_ARENA_MAX=1
Summary: Start ovirt-engine and ovirt-engine-notifier services by default with MALLOC_...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Backend.Core
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-4.2.0
: ---
Assignee: Martin Perina
QA Contact: Radim Hrazdil
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-23 15:17 UTC by Roy Golan
Modified: 2017-12-20 10:42 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-12-20 10:42:51 UTC
oVirt Team: Infra
Embargoed:
rule-engine: ovirt-4.2+
lsvaty: testing_plan_complete-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 83578 0 master MERGED services: Set MALLOC_ARENA_MAX=1 for ovirt-engine JVM 2017-11-06 10:02:38 UTC
oVirt gerrit 83579 0 master MERGED services: Set MALLOC_ARENA_MAX=1 for notifier JVM 2017-11-06 10:02:41 UTC

Description Roy Golan 2017-10-23 15:17:46 UTC
After I observed large amount of virtual memory allocated by ovirt-engine, I found about MALLOC_MAX_ARENA and got to this wildfly ticket[1]
which decided to start wildfly with default of 1.

Unfortunately we don't use standlone.sh, so we miss those valuable additions.

This change won't affect performance, but on machines with large amount of cores it will definitely mean less virtual memory allocation.

I suggest going with what wildfly does by default and add this to our services/ovirt-engine.conf

 MALLOC_ARENA_MAX=1


Another approach would be to somehow use standalone.sh so we gain all the changes for free - but that of course is not free, it means a change to our home brew ovirt-engine.py.


[1]https://issues.jboss.org/browse/WFCORE-2959

Version-Release number of selected component (if applicable):
all

How reproducible:
100%

Steps to Reproduce:
run ovirt-engine with MALLOC_ARENA_MAX set to 128 (to emulate a machine with 16 cores) and then 1, check RSS and Virtual memory using
ps  -o vsize,rss -p $(pidof ovirt-engine)

Comment 1 Roy Golan 2017-10-24 13:13:44 UTC
Moreover, high amount of virtual memory, will affect the oom_score of the ovirt-engine process, leading to it be even more likely be killed by oom. 

I can recall some reports to the mailing list about oom, maybe that can explain some of them.

Comment 2 Martin Perina 2017-11-03 10:39:21 UTC
The same optimization can be also used for ovirt-engine-notifier

Comment 3 Roy Golan 2017-11-14 08:05:23 UTC
(In reply to Martin Perina from comment #2)
> The same optimization can be also used for ovirt-engine-notifier

Might be. If the number of thread the notifier uses is small, which I assume it is, then it won't have an effect, but you are familiar with it more than I.

Comment 4 Radim Hrazdil 2017-11-16 14:51:18 UTC
Verified that value of ENGINE_MALLOC_ARENA_MAX in /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf is by default set to 1.

Comment 5 Sandro Bonazzola 2017-12-20 10:42:51 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.