RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2093847 - [RFE] Automate determining initial cpu isolation setup and update tuned-profiles-realtime
Summary: [RFE] Automate determining initial cpu isolation setup and update tuned-profi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: tuned
Version: 9.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jaroslav Škarvada
QA Contact: Robin Hack
Šárka Jana
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-06 08:11 UTC by Juri Lelli
Modified: 2022-11-15 13:05 UTC (History)
18 users (show)

Fixed In Version: tuned-2.19.0-0.1.rc1.el9
Doc Type: Enhancement
Doc Text:
.TuneD real-time profiles now auto determine initial CPU isolation setup TuneD is a service for monitoring your system and optimizing the performance profile. You can also isolate central processing units (CPUs) using the `tuned-profiles-realtime` package to give application threads the most execution time possible. Previously, the real-time profiles for systems running the real-time kernel did not load if you did not specify the list of CPUs to isolate in the `isolated_cores` parameter. With this enhancement, TuneD introduces the `calc_isolated_cores` built-in function that automatically calculates housekeeping and isolated cores lists, and applies the calculation to the `isolated_cores` parameter. With the automatic preset, one core from each socket is reserved for housekeeping, and you can start using the real-time profile without any additional steps. If you want to change the preset, customize the `isolated_cores` parameter by specifying the list of CPUs to isolate.
Clone Of:
Environment:
Last Closed: 2022-11-15 11:17:00 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-124323 0 None None None 2022-06-06 08:17:29 UTC
Red Hat Product Errata RHBA-2022:8321 0 None None None 2022-11-15 11:17:07 UTC

Description Juri Lelli 2022-06-06 08:11:33 UTC 
Description of problem:
realtime tuned profile can't be used right away after install since it
requires a valid 'isolated_cores' parameter from realtime-variables.conf.

What we (RT team and RT-QE team) do for testing (gating) kernels is to
define it as 'isolated_cores=' (void) so that the additional tuning done by
applying the profile is performed (disabling watchdogs, intel_pstate, etc.).

This has been OK so far for relatively small single-socket systems, but it
turns out that:

1. Larger (dual+ sockets) boxes can experience high latencies w/o isolation
2. Most of RT customers don't of course run in production w/o some kind of isolation
3. (minor) It might be convenient for users and automation to have the realtime
   profile available to use right after install w/o any additional steps

It looks like a sane default (as possibly a base for further tuning) is to use
1 or 2 CPUs per socket as housekeeping and isolate the rest of the CPUs for the
actual workload. So, e.g., in a system with 2 sockets and 40 cores per socket
(socket#0 [0-39] socket#1 [40-79]) a sane initial default would be to configure
'isolated_cores=2-39,42-79'.

Request for this RFE is to implement logic to automatically inspect a system's
topology (using for example lstopo-no-graphics), calculate housekeeping and
isolated cores lists and apply the calculation to isolated_cores parameter.
With this automated process in place a user can then simply start the profile
and run tests on the system w/o any initial need for hands on tuning.

Version-Release number of selected component (if applicable):
Possibly implement this in 9.2, so that the feature can be used for kernel-rt
gating tests and RHEL9-RT certifications.
Comment 2 Luiz Capitulino 2022-06-23 20:33:41 UTC 
Hi Juri,

This looks like a great idea!

Do we know if/how this will affect the virtual real-time profiles? Would be great if you or Jarda include the Virt-RT team in the patches so that we can review and test them with the KVM-RT test-cases.
Comment 5 Jaroslav Škarvada 2022-08-08 21:46:08 UTC 
Upstream PR:
https://github.com/redhat-performance/tuned/pull/453
Comment 6 Jaroslav Škarvada 2022-08-08 21:52:29 UTC 
(In reply to Luiz Capitulino from comment #2)
> Hi Juri,
> 
> This looks like a great idea!
> 
> Do we know if/how this will affect the virtual real-time profiles? Would be
> great if you or Jarda include the Virt-RT team in the patches so that we can
> review and test them with the KVM-RT test-cases.

IMHO the difference is:
- previously the stock realtime profiles refused to load if user haven't specified the isolated_cores
- now, with the patch, it presets isolated_cores according to the comment 0

Users can still customize the isolated_cores (change what was preset for her/him). In case she/he already customized it, nothing will change for her/him.

I preset one housekeeping core per socket in the patch. If two are needed, please let me know to update the patch.
Comment 26 Jaroslav Škarvada 2022-11-01 13:46:10 UTC 
With this enhancement, TuneD introduces the `calc_isolated_cores` built-in function that automatically calculates housekeeping and isolated cores lists, and applies the calculation to the `isolated_cores` parameter. With the automatic preset, one core from each socket is reserved for housekeeping, and you can start using the real-time profile without any additional steps.
If you want to change the preset, customize the `isolated_cores` parameter by specifying the list of CPUs to isolate.

Example:
isolated_cores=${f:calc_isolated_cores:2}

On machine with 2 sockets, each 4 cores, it will expand to:
isolated_cores=2, 3, 6, 7

I.e. cores 0, 1 and 4, 5 will be used for housekeeping.
Comment 30 errata-xmlrpc 2022-11-15 11:17:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (tuned bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8321
Note You need to log in before you can comment on or make changes to this bug.