Bug 1118734

Summary:	RFE: compute the value for smp_mflags from number of cpus and available memory
Product:	[Fedora] Fedora	Reporter:	Dan Horák <dan>
Component:	rpm	Assignee:	Panu Matilainen <pmatilai>
Status:	CLOSED RAWHIDE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	debarshir, fedoraproject, jonathan, ktdreyer, mcatanza, mjw, packaging-team-maint, pmatilai, pmoravco, tpopela, vmukhame
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-07-04 10:01:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dan Horák 2014-07-11 12:24:55 UTC

I would like to suggest to change how the value for smp_mflags is computed. Currently it is the number of cpus online and 16 is the max. I suggest to include the memory size into the calculation, because with for example C++ code the compiler can easily require 2GB per process. So if the builder machine doesn't have adequate memory it starts swapping and later the compiler (or linker) gets killed by the kernel OOM killer.

my proposal
smp_mflags = MIN( getconf _NPROCESSORS_ONLN , RAMSIZE / TASKSIZE)
and use eg. TASKSIZE=2GB

The real case where OOM is in action during builds is on ppc64 builders with 48 (shared virtual) CPUs and 10GB of memory (+ some swap) and building ceph. I think it can also help on ARM with 4 CPUs and only 4GB of memory when they start swapping easily.

Comment 1 Fedora Admin XMLRPC Client 2014-11-14 07:24:40 UTC

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 2 Panu Matilainen 2019-03-07 10:46:50 UTC

%_smp_mflags lives in rpm now, switching component (wow this is an old one...)

It'd need some tunables (tasksize at least) and maybe 32/64bit-ness should be taken into account for the heuristics too.

Comment 3 Dan Horák 2019-08-05 07:50:42 UTC

Ack, tunable "tasksize" makes sense.

Comment 4 Panu Matilainen 2019-08-28 11:32:38 UTC

FWIW, there's a work-in-progress PR at upstream where further input (+ testing) is welcome:
https://github.com/rpm-software-management/rpm/pull/821

Comment 5 Tomas Popela 2020-06-08 10:44:30 UTC

We've been hit by this recently in RHEL again with Thunderbird. The builder hard 8 CPUs, but just 14 GB of available RAM.

We will start to use what was suggested by Dan in comment 0 for Thunderbird and Firefox:

# Free memory in kB
MEM_AVAILABLE=$(grep MemAvailable /proc/meminfo | awk '{ print $2 }')
# Usually the compiler processes can take 2 GB of memory at peaks
TASK_SIZE=2000000
MEM_CONSTRAINED_JOBS=$(( MEM_AVAILABLE / TASK_SIZE ))
CPU_AVAILABLE=$(/usr/bin/getconf _NPROCESSORS_ONLN)
# Pick the minimum from available CPUs or memory constrained number of jobs
MOZ_SMP_FLAGS=$([ "$CPU_AVAILABLE" -le "$MEM_CONSTRAINED_JOBS" ] && echo "$CPU_AVAILABLE" || echo "$MEM_CONSTRAINED_JOBS")

Comment 6 Panu Matilainen 2023-07-04 10:01:17 UTC

This is (finally) implemented in rpm 4.19.