Bug 1663412

Summary:	Tuned takes too long to reload/start when "ulimit -n" is high.
Product:	Red Hat Enterprise Linux 7	Reporter:	Jiří Mencák <jmencak>
Component:	tuned	Assignee:	Ondřej Lysoněk <olysonek>
Status:	CLOSED ERRATA	QA Contact:	Robin Hack <rhack>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	7.7-Alt	CC:	jeder, jmencak, jskarvad, olysonek, psklenar, rhack
Target Milestone:	rc	Keywords:	Patch, Upstream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tuned-2.11.0-0.1.rc1.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1666678 (view as bug list)		Environment:
Last Closed:	2019-08-06 13:04:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1666678

Description Jiří Mencák 2019-01-04 08:42:30 UTC

Description of problem:
When the soft system limit for number of open files is high (e.g. 1M), it can take ~20 seconds to start/reload tuned.  The problem is caused by the python subprocess.Popen(... close_fds=True ...) call when running multiple sysctl commands, each of which can take roughly 0.5s.

Version-Release number of selected component (if applicable):
All.

How reproducible:
Always when setting a profile using sysctl plugin with multiple values, e.g. through profile inheritance.

Steps to Reproduce:
1. systemctl stop tuned
2. ulimit -Sn 1048576
3. tuned --debug

Actual results:
An openshift-control-plane profile reload/start takes ~20s

<snip>
2019-01-04 03:38:14,995 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'vm.swappiness=30'].
2019-01-04 03:38:15,570 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-e', 'net.netfilter.nf_conntrack_max']
2019-01-04 03:38:15,570 DEBUG    tuned.utils.commands: Executing ['sysctl', '-e', 'net.netfilter.nf_conntrack_max'].
2019-01-04 03:38:16,106 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-q', '-w', 'net.netfilter.nf_conntrack_max=1048576']
2019-01-04 03:38:16,106 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'net.netfilter.nf_conntrack_max=1048576'].
2019-01-04 03:38:16,634 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh2']
2019-01-04 03:38:16,634 DEBUG    tuned.utils.commands: Executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh2'].
2019-01-04 03:38:17,246 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh2=32768']
2019-01-04 03:38:17,246 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh2=32768'].
2019-01-04 03:38:17,887 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh3']
2019-01-04 03:38:17,887 DEBUG    tuned.utils.commands: Executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh3'].
2019-01-04 03:38:18,478 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh3=65536']
2019-01-04 03:38:18,478 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh3=65536'].

</snip>

Expected results:
A full reload/start takes ~0.5s

Additional info:
https://github.com/redhat-performance/tuned/issues/146

Comment 2 Ondřej Lysoněk 2019-01-14 14:20:55 UTC

(In reply to jmencak from comment #0)
> Steps to Reproduce:
> 1. systemctl stop tuned
> 2. ulimit -Sn 1048576
> 3. tuned --debug

Do you run tuned directly like this, or do you run it using 'systemctl start'? If you run it using systemd, we could just add 'LimitNOFILE=' to the systemd service. Would that work for you?

Comment 3 Ondřej Lysoněk 2019-01-14 14:45:33 UTC

To be clear, I still intend to change tuned not to execute sysctl for each setting, as discussed in the upstream issue. In fact, I've already implemented that:
https://github.com/olysonek/tuned/commit/001d2a3c49a268ef9abb7d03a57ec22100266fdb

Comment 4 Jiří Mencák 2019-01-14 15:03:35 UTC

At the moment tuned is run outside of systemd inside a pod.  Currently using a workaround by lowering the number open files: https://github.com/openshift/openshift-tuned/blob/master/assets/bin/run#L13

Comment 5 Ondřej Lysoněk 2019-02-25 15:00:46 UTC

(In reply to jmencak from comment #4)
> At the moment tuned is run outside of systemd inside a pod.  Currently using
> a workaround by lowering the number open files:
> https://github.com/openshift/openshift-tuned/blob/master/assets/bin/run#L13

Ok, given that you already have a workaround in place, I think it's not necessary to add more workarounds to Tuned. So we won't change the Tuned service.

The underlying problem should be fixed in the following pull request:
https://github.com/redhat-performance/tuned/pull/160

Comment 10 errata-xmlrpc 2019-08-06 13:04:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2201