Bug 1663412 - Tuned takes too long to reload/start when "ulimit -n" is high.
Summary: Tuned takes too long to reload/start when "ulimit -n" is high.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned
Version: 7.7-Alt
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Ondřej Lysoněk
QA Contact: Robin Hack
URL:
Whiteboard:
Depends On:
Blocks: 1666678
TreeView+ depends on / blocked
 
Reported: 2019-01-04 08:42 UTC by jmencak
Modified: 2019-08-06 13:05 UTC (History)
6 users (show)

Fixed In Version: tuned-2.11.0-0.1.rc1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1666678 (view as bug list)
Environment:
Last Closed: 2019-08-06 13:04:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github redhat-performance tuned pull 160 None None None 2019-02-25 14:59:29 UTC
Red Hat Product Errata RHBA-2019:2201 None None None 2019-08-06 13:05:18 UTC

Description jmencak 2019-01-04 08:42:30 UTC
Description of problem:
When the soft system limit for number of open files is high (e.g. 1M), it can take ~20 seconds to start/reload tuned.  The problem is caused by the python subprocess.Popen(... close_fds=True ...) call when running multiple sysctl commands, each of which can take roughly 0.5s.

Version-Release number of selected component (if applicable):
All.

How reproducible:
Always when setting a profile using sysctl plugin with multiple values, e.g. through profile inheritance.

Steps to Reproduce:
1. systemctl stop tuned
2. ulimit -Sn 1048576
3. tuned --debug

Actual results:
An openshift-control-plane profile reload/start takes ~20s

<snip>
2019-01-04 03:38:14,995 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'vm.swappiness=30'].
2019-01-04 03:38:15,570 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-e', 'net.netfilter.nf_conntrack_max']
2019-01-04 03:38:15,570 DEBUG    tuned.utils.commands: Executing ['sysctl', '-e', 'net.netfilter.nf_conntrack_max'].
2019-01-04 03:38:16,106 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-q', '-w', 'net.netfilter.nf_conntrack_max=1048576']
2019-01-04 03:38:16,106 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'net.netfilter.nf_conntrack_max=1048576'].
2019-01-04 03:38:16,634 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh2']
2019-01-04 03:38:16,634 DEBUG    tuned.utils.commands: Executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh2'].
2019-01-04 03:38:17,246 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh2=32768']
2019-01-04 03:38:17,246 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh2=32768'].
2019-01-04 03:38:17,887 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh3']
2019-01-04 03:38:17,887 DEBUG    tuned.utils.commands: Executing ['sysctl', '-e', 'net.ipv4.neigh.default.gc_thresh3'].
2019-01-04 03:38:18,478 DEBUG    tuned.plugins.plugin_sysctl: executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh3=65536']
2019-01-04 03:38:18,478 DEBUG    tuned.utils.commands: Executing ['sysctl', '-q', '-w', 'net.ipv4.neigh.default.gc_thresh3=65536'].

</snip>

Expected results:
A full reload/start takes ~0.5s

Additional info:
https://github.com/redhat-performance/tuned/issues/146

Comment 2 Ondřej Lysoněk 2019-01-14 14:20:55 UTC
(In reply to jmencak from comment #0)
> Steps to Reproduce:
> 1. systemctl stop tuned
> 2. ulimit -Sn 1048576
> 3. tuned --debug

Do you run tuned directly like this, or do you run it using 'systemctl start'? If you run it using systemd, we could just add 'LimitNOFILE=' to the systemd service. Would that work for you?

Comment 3 Ondřej Lysoněk 2019-01-14 14:45:33 UTC
To be clear, I still intend to change tuned not to execute sysctl for each setting, as discussed in the upstream issue. In fact, I've already implemented that:
https://github.com/olysonek/tuned/commit/001d2a3c49a268ef9abb7d03a57ec22100266fdb

Comment 4 jmencak 2019-01-14 15:03:35 UTC
At the moment tuned is run outside of systemd inside a pod.  Currently using a workaround by lowering the number open files: https://github.com/openshift/openshift-tuned/blob/master/assets/bin/run#L13

Comment 5 Ondřej Lysoněk 2019-02-25 15:00:46 UTC
(In reply to jmencak from comment #4)
> At the moment tuned is run outside of systemd inside a pod.  Currently using
> a workaround by lowering the number open files:
> https://github.com/openshift/openshift-tuned/blob/master/assets/bin/run#L13

Ok, given that you already have a workaround in place, I think it's not necessary to add more workarounds to Tuned. So we won't change the Tuned service.

The underlying problem should be fixed in the following pull request:
https://github.com/redhat-performance/tuned/pull/160

Comment 10 errata-xmlrpc 2019-08-06 13:04:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2201


Note You need to log in before you can comment on or make changes to this bug.