RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1957829 - [RHEL-9] - Please remove all kernel.sched settings (except for kernel.sched_rt_runtime_us) from tuned profiles
Summary: [RHEL-9] - Please remove all kernel.sched settings (except for kernel.sched_r...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: tuned
Version: 9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: beta
: ---
Assignee: Jaroslav Škarvada
QA Contact: Robin Hack
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-06 14:49 UTC by Jiri Hladky
Modified: 2021-12-07 21:45 UTC (History)
9 users (show)

Fixed In Version: tuned-2.16.0-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-07 21:42:00 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Jiri Hladky 2021-05-06 14:49:54 UTC 
For RHEL-9, we have evaluated the performance impact of kernel.sched settings in tuned profiles and we have reached a conclusion that elevated settings are no longer needed. 

Testing was extensive, performed by both Shak's group and Kernel Performance QE team, and was discussed on several kscale meetings. 

There is an agreement to remove all kernel.sched settings (except for kernel.sched_rt_runtime_us) from tuned profiles and use kernel defaults instead. 

Could you please update the tuned profiles accordingly? 

Thanks a lot
Jirka

PS:

This command can be used to check the settings:

$find /usr/lib/tuned/ -name tuned.conf -exec grep -H kernel.sched {} \; | grep -v kernel.sched_rt_runtime_us

When this is implemented, the above command should return an empty list.
Comment 1 Joe Mario 2021-05-06 15:54:04 UTC 
Hi Jirka and Jaroslav:
We should probably only focus on the tuned profiles that we've been testing and are closely coupled to RHEL.
When I look at the list of tuned profiles in https://github.com/redhat-performance/tuned that set the kernel.sched_* knobs, I see the following:

 kernel.sched_migration_cost_ns = 5000000
     1590      4 -rw-rw-r--   1 jmario   jmario        524 Apr 30 13:50 tuned-master/profiles/virtual-host/tuned.conf

 kernel.sched_min_granularity_ns = 10000000
 kernel.sched_wakeup_granularity_ns = 15000000
 kernel.sched_migration_cost_ns=5000000
     1588      4 -rw-rw-r--   1 jmario   jmario       2892 Apr 30 13:50 tuned-master/profiles/throughput-performance/tuned.conf

 kernel.sched_min_granularity_ns = 10000000
 kernel.sched_wakeup_granularity_ns = 15000000
     1585      4 -rw-rw-r--   1 jmario   jmario        600 Apr 30 13:50 tuned-master/profiles/spectrumscale-ece/tuned.conf

 kernel.sched_min_granularity_ns = 3000000
 kernel.sched_wakeup_granularity_ns = 4000000
     1582      4 -rw-rw-r--   1 jmario   jmario        434 Apr 30 13:50 tuned-master/profiles/sap-hana/tuned.conf

 kernel.sched_min_granularity_ns=10000000
 kernel.sched_migration_cost_ns=50000000
 kernel.sched_autogroup_enabled = 0
     1570      4 -rw-rw-r--   1 jmario   jmario       1890 Apr 30 13:50 tuned-master/profiles/postgresql/tuned.conf

 kernel.sched_latency_ns=60000000
 kernel.sched_min_granularity_ns=15000000
 kernel.sched_wakeup_granularity_ns=2000000
     1565      4 -rw-rw-r--   1 jmario   jmario        316 Apr 30 13:50 tuned-master/profiles/mssql/tuned.conf

 kernel.sched_min_granularity_ns=3000000
 kernel.sched_wakeup_granularity_ns=4000000
 kernel.sched_migration_cost_ns=5000000
     1564      4 -rw-rw-r--   1 jmario   jmario       1604 Apr 30 13:50 tuned-master/profiles/latency-performance/tuned.conf

 kernel.sched_autogroup_enabled=1
     1556      4 -rw-rw-r--   1 jmario   jmario        136 Apr 30 13:50 tuned-master/profiles/desktop/tuned.conf

 kernel.sched_min_granularity_ns = 10000000
 kernel.sched_wakeup_granularity_ns = 15000000
     1546      4 -rw-rw-r--   1 jmario   jmario       2126 Apr 30 13:50 tuned-master/profiles/accelerator-performance/tuned.conf

The testing we've been doing has been with the following (or the profiles that inherit the following):
 - virtual-host
 - throughput-performance
 - latency-performance

The sap-hana profile (above) actually restores the two sched flags back to the kernel defaults, so they can probably be removed, but we should have Dave D. weigh in on this.  I recall the flags behaved differently depending on whether HANA was run on the host or in a guest.

The mmsql profile should stay as it is.  We didn't define the contents in it. The MSSql perf team at Microsoft did that.  They need to bless any changes first.

I'm thinking we should keep the above sched_autogroup_enabled flags, as we didn't test them.

We didn't do any testing with the accelerator-performance profile.  It may be good to not touch that profile and let the owner of it do his/her own testing.

I'm adding Sanjay so he can weigh in on his testing with the postgresql profile.  I thought he did his testing with throughput-performance, but I'll let him confirm.

Thanks for getting this going.
Joe
Comment 2 Jiri Hladky 2021-05-06 18:12:31 UTC 
Thank you for narrowing this down, Joe!
Comment 3 djdumas 2021-05-06 19:44:28 UTC 
(In reply to Joe Mario from comment #1)
> Hi Jirka and Jaroslav:
> We should probably only focus on the tuned profiles that we've been testing
> and are closely coupled to RHEL.
> When I look at the list of tuned profiles in
> https://github.com/redhat-performance/tuned that set the kernel.sched_*
> knobs, I see the following:
> 
>  kernel.sched_migration_cost_ns = 5000000
>      1590      4 -rw-rw-r--   1 jmario   jmario        524 Apr 30 13:50
> tuned-master/profiles/virtual-host/tuned.conf
Hmm, I use virtual-host in all my HANA KVM testing but forgot to test the impact of this when run as kernel default - will check it Joe

<snip>
 
>  kernel.sched_min_granularity_ns = 3000000
>  kernel.sched_wakeup_granularity_ns = 4000000
>      1582      4 -rw-rw-r--   1 jmario   jmario        434 Apr 30 13:50
> tuned-master/profiles/sap-hana/tuned.conf

Yes we went to these values for HANA around the RHEL 8.0 or 8.1 timeframe I think - for performance reasons - and they are the kernel defaults

<snip>
 
> The testing we've been doing has been with the following (or the profiles
> that inherit the following):
>  - virtual-host
>  - throughput-performance
>  - latency-performance
> 
> The sap-hana profile (above) actually restores the two sched flags back to
> the kernel defaults, so they can probably be removed, but we should have
> Dave D. weigh in on this.  I recall the flags behaved differently depending
> on whether HANA was run on the host or in a guest.

Using sap-hana profile as shown above with kernel defaults results in the best bare metal performance.

There is a difference (up to ~3%) when running HANA in a guest (RHEL 9 on host and guest) and using higher values for kernel.sched_min_granularity_ns and kernel.sched_wakeup_granularity_ns. The settings being tested are those listed by Joe M in https://bugzilla.redhat.com/show_bug.cgi?id=1861734
Still have a bit more work to do in this area.

Thanks,
Dave
Comment 4 Joe Mario 2021-05-07 22:06:50 UTC 
Update on two profiles:

For the postgresql tuned profile, after talking with Sanjay, we agreed the values in that profile were set by the postgresql maintainers on benchmarks and systems they run.  The right thing to do would be to let them decide if they want the sched_* values changed in their profile.

For the mssql profile, we've been communicating with the MSsql team that defined those values.  They will be retesting and will let us know if they want any changes.  
They did say the values in the mssql profile were based on their RHEL-7 testing, and that they noticed the values don't seem to be optimal on RHEL-8.

In summary, Jaroslav, don't touch the postgresql or mssql profiles, along with the previously mentioned accelerator-performance profile.

Joe
Comment 5 Joe Mario 2021-06-08 14:27:51 UTC 
Hi Jaroslav:

Both Jirka Hladky and our teams have done a lot of RHEL-9 performance testing with the following flags removed from the named tuned profiles.  We've determined the flags can be removed, as noted below.

From the virtual-host profile, please remove:
  kernel.sched_migration_cost_ns = 5000000

From the throughput-performance profile, please remove:
  kernel.sched_min_granularity_ns = 10000000
  kernel.sched_wakeup_granularity_ns = 15000000
  kernel.sched_migration_cost_ns=5000000

From the sap-hana profile, please remove:
  kernel.sched_min_granularity_ns = 3000000
  kernel.sched_wakeup_granularity_ns = 4000000

From the latency-performance profile, please remove:
  kernel.sched_min_granularity_ns=3000000
  kernel.sched_wakeup_granularity_ns=4000000
  kernel.sched_migration_cost_ns=5000000

There are other profiles that are setting kernel.sched_* flags.  We do not maintain those profiles and therefore should not be removing any flags from them.

Let us know if you have any questions.
Joe
Comment 6 Jaroslav Škarvada 2021-06-11 18:47:03 UTC 
(In reply to Joe Mario from comment #5)
> Hi Jaroslav:
> 
> Both Jirka Hladky and our teams have done a lot of RHEL-9 performance
> testing with the following flags removed from the named tuned profiles. 
> We've determined the flags can be removed, as noted below.
> 
> From the virtual-host profile, please remove:
>   kernel.sched_migration_cost_ns = 5000000
> 
> From the throughput-performance profile, please remove:
>   kernel.sched_min_granularity_ns = 10000000
>   kernel.sched_wakeup_granularity_ns = 15000000
>   kernel.sched_migration_cost_ns=5000000
> 
> From the sap-hana profile, please remove:
>   kernel.sched_min_granularity_ns = 3000000
>   kernel.sched_wakeup_granularity_ns = 4000000
> 
> From the latency-performance profile, please remove:
>   kernel.sched_min_granularity_ns=3000000
>   kernel.sched_wakeup_granularity_ns=4000000
>   kernel.sched_migration_cost_ns=5000000
> 
> There are other profiles that are setting kernel.sched_* flags.  We do not
> maintain those profiles and therefore should not be removing any flags from
> them.
> 
> Let us know if you have any questions.
> Joe

Thanks for info.
Comment 7 Jaroslav Škarvada 2021-06-29 22:26:29 UTC 
Currently, the TuneD engine and profiles are the same for both RHEL-8 and RHEL-9. I will drop the sched tuning in upstream and in RHEL-9. I will keep it on RHEL-8 by utilizing downstream RHEL-8 patches. But if you want to drop it also on RHEL-8, please let me know - it would simplify maintenance.
Comment 8 Jaroslav Škarvada 2021-08-24 11:17:12 UTC 
Upstream commit:
https://github.com/redhat-performance/tuned/pull/365/commits/c6d6fdcc4c944df9998e0ebe75f31cc8aed452c1
Note You need to log in before you can comment on or make changes to this bug.