Bug 1505452

Summary:	Start ovirt-engine and ovirt-engine-notifier services by default with MALLOC_ARENA_MAX=1
Product:	[oVirt] ovirt-engine	Reporter:	Roy Golan <rgolan>
Component:	Backend.Core	Assignee:	Martin Perina <mperina>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Radim Hrazdil <rhrazdil>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.2.0	CC:	bugs, lsvaty, mperina, pstehlik
Target Milestone:	ovirt-4.2.0	Keywords:	Improvement, Performance
Target Release:	---	Flags:	rule-engine: ovirt-4.2+ lsvaty: testing_plan_complete-
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-20 10:42:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Roy Golan 2017-10-23 15:17:46 UTC

After I observed large amount of virtual memory allocated by ovirt-engine, I found about MALLOC_MAX_ARENA and got to this wildfly ticket[1]
which decided to start wildfly with default of 1.

Unfortunately we don't use standlone.sh, so we miss those valuable additions.

This change won't affect performance, but on machines with large amount of cores it will definitely mean less virtual memory allocation.

I suggest going with what wildfly does by default and add this to our services/ovirt-engine.conf

 MALLOC_ARENA_MAX=1


Another approach would be to somehow use standalone.sh so we gain all the changes for free - but that of course is not free, it means a change to our home brew ovirt-engine.py.


[1]https://issues.jboss.org/browse/WFCORE-2959

Version-Release number of selected component (if applicable):
all

How reproducible:
100%

Steps to Reproduce:
run ovirt-engine with MALLOC_ARENA_MAX set to 128 (to emulate a machine with 16 cores) and then 1, check RSS and Virtual memory using
ps  -o vsize,rss -p $(pidof ovirt-engine)

Comment 1 Roy Golan 2017-10-24 13:13:44 UTC

Moreover, high amount of virtual memory, will affect the oom_score of the ovirt-engine process, leading to it be even more likely be killed by oom. 

I can recall some reports to the mailing list about oom, maybe that can explain some of them.

Comment 2 Martin Perina 2017-11-03 10:39:21 UTC

The same optimization can be also used for ovirt-engine-notifier

Comment 3 Roy Golan 2017-11-14 08:05:23 UTC

(In reply to Martin Perina from comment #2)
> The same optimization can be also used for ovirt-engine-notifier

Might be. If the number of thread the notifier uses is small, which I assume it is, then it won't have an effect, but you are familiar with it more than I.

Comment 4 Radim Hrazdil 2017-11-16 14:51:18 UTC

Verified that value of ENGINE_MALLOC_ARENA_MAX in /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf is by default set to 1.

Comment 5 Sandro Bonazzola 2017-12-20 10:42:51 UTC

This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.