RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1451073 - tuned: use skew_tick=1 in all low-latency profiles
Summary: tuned: use skew_tick=1 in all low-latency profiles
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jaroslav Škarvada
QA Contact: Tereza Cerna
URL:
Whiteboard:
Depends On:
Blocks: 1465887 TUNED-7.5-REBASE 1485946
TreeView+ depends on / blocked
 
Reported: 2017-05-15 17:11 UTC by Luiz Capitulino
Modified: 2021-06-10 12:19 UTC (History)
9 users (show)

Fixed In Version: tuned-2.9.0-0.1.rc1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 16:01:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0879 0 normal SHIPPED_LIVE tuned bug fix and enhancement update 2018-04-10 13:39:08 UTC

Description Luiz Capitulino 2017-05-15 17:11:50 UTC 
Description of problem:

On Linux by default the tick timer is programmed to fire at the same time on all CPUs. This may cause lock contention as some locks are taken by the tick handler, as the jiffies_lock and an RCU lock.

The skew_tick=1 parameter causes the kernel to program each CPU's tick timer to fire at different times, avoiding any possible lock contention. Kernel developers advise passing skew_tick=1 to the kernel when the workload is very sensible to jittier.

Due to this reason, we're considering using skew_tick=1 in the network-latency profile, which will cause it to be inherited by the cpu-partitioning profile and realtime profiles (although the realtime profiles are already getting skew_tick support see bug 1447938).

This BZ should gather data in favour or against using skew_tick=1.


Version-Release number of selected component (if applicable): tuned-2.8.0-3.el7.noarch
Comment 2 Luiz Capitulino 2017-05-15 17:18:42 UTC 
I'll do the investigatin and post the patches in case we decide to use skew_tick=1, so I'm taking the BZ.
Comment 3 Joe Mario 2017-05-16 13:47:12 UTC 
A few thoughts:

 - Current thinking is skew_tick=1 is only relevant when nohz_full is NOT used.
   Though it should not hurt if used with nohz_full.

 - Do we know of any downside to skewing the timer ticks?  I know of none, 
   but since this is not a widely used flag, it's worth understanding.

Joe
Comment 4 Luiz Capitulino 2017-05-16 18:33:03 UTC 
The only known downside is that skew_tick=1 could theoretically cause higher power consumption because a socket would take more time (or be unable) to shut itself down when idle. However, this hasn't been measured on recent kernels.

Regarding nohz_full, I guess you may be referring to bug 1404440. In this bug, sysjitter behavior with nohz_full cause so much jitter that skew_tick=1 is probably not going to make much difference. However, it's not clear if this behavior is specific to sysjitter. This means that there might be cases where skew_tick=1 works well with nohz_full= (I expect this to be the case with RT for example, although I haven't checked that yet).
Comment 5 Joe Mario 2017-05-16 18:44:44 UTC 
Hi Luiz:
 Your reply made me realize the error in my thinking for my bullet 1 above.

 I was thinking nohz_full brings the ticks down to 1/sec, which negates the risk of the tick thundering herd problem.

 But if there is more than one thread on the cpu, then there will be 1000 ticks/sec (which I forgot in my comment-3 reply.

 So yes do include skew_tick=1 when nohz_full is used.

Joe
Comment 6 Jaroslav Škarvada 2017-05-29 15:51:51 UTC 
Upstream commit fixing the problem:
https://github.com/redhat-performance/tuned/commit/9bc020df37d49b065aac0b164f0cc3ec254561a5
Comment 7 Jaroslav Škarvada 2017-06-02 09:18:20 UTC 
(In reply to Jaroslav Škarvada from comment #6)
> Upstream commit fixing the problem:
> https://github.com/redhat-performance/tuned/commit/
> 9bc020df37d49b065aac0b164f0cc3ec254561a5

The mentioned upstream commit is for RT profiles only.
Comment 8 Luiz Capitulino 2017-06-02 13:18:13 UTC 
Yes, this one is not a blocker and is not being fixed yet. Moving to 7.5.
Comment 10 Jaroslav Škarvada 2017-07-19 11:55:42 UTC 
Fixed by the following upstream commit:
https://github.com/redhat-performance/tuned/commit/643864f10cbf2891d8fd2b4c33ec358904ff209b
Comment 11 Luiz Capitulino 2017-07-19 12:56:12 UTC 
Jaroslav,

The cpu-partitioning profile defines "cmdline" in the bootloader section. Shouldn't you change it as you did in commit 4483641602? Btw, I didn't check if the other profiles including network-latency define cmdline too.
Comment 12 Jaroslav Škarvada 2017-07-19 13:08:44 UTC 
(In reply to Luiz Capitulino from comment #11)
> Jaroslav,
> 
> The cpu-partitioning profile defines "cmdline" in the bootloader section.
> Shouldn't you change it as you did in commit 4483641602? Btw, I didn't check
> if the other profiles including network-latency define cmdline too.

Thanks, it would miss skew_tick=1 in cpu_partitioning. Fixed by:
https://github.com/redhat-performance/tuned/commit/d1379afbec81a24e3c3c8444654cd6e00f0eb96d

AFAIK there are no more profiles including it.
Comment 13 Luiz Capitulino 2017-07-19 13:10:24 UTC 
Excellent, thanks!
Comment 14 Jaroslav Škarvada 2017-07-19 13:11:47 UTC 
(In reply to Jaroslav Škarvada from comment #12)
> AFAIK there are no more profiles including it.

Not counting the realtime which I already fixed in the previous commit.
Comment 15 Luiz Capitulino 2017-10-03 20:51:30 UTC 
The cpu-partitioning profile needs skew_tick=1 for 7.4.z. If this change is too complex to backport, we can just set "skew_tick=1" in the bootloader section for cpu-partitioning.
Comment 18 Fedora Update System 2017-10-13 14:20:46 UTC 
tuned-2.9.0-0.1.rc1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df
Comment 20 Fedora Update System 2017-10-13 22:25:11 UTC 
tuned-2.9.0-0.1.rc1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df
Comment 21 Fedora Update System 2017-10-13 23:24:59 UTC 
tuned-2.9.0-0.1.rc1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5f0849d207
Comment 22 Fedora Update System 2017-10-29 21:07:00 UTC 
tuned-2.9.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0e45ce4685
Comment 23 Fedora Update System 2017-10-29 21:13:07 UTC 
tuned-2.9.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c30e9bd1ea
Comment 24 Luiz Capitulino 2018-01-22 22:02:10 UTC 
I verified this BZ. I checked that skew_tick=1 option is passed to the kernel when using profiles: cpu-partitioning, network-latency, realtime, realtime-virtual-host and realtime-virtual-guest.
Comment 25 Tereza Cerna 2018-01-26 12:39:00 UTC 
New test case for this bugzilla was created:
/CoreOS/tuned/Regression/bz1451073-use_skew_tick_in_cmdline

======================================================
Verified in:
    tuned-2.9.0-1.el7.noarch
    tuned-profiles-realtime-2.9.0-1.el7.noarch
    tuned-profiles-nfv-host-2.9.0-1.el7.noarch
    tuned-profiles-nfv-guest-2.9.0-1.el7.noarch
    tuned-profiles-cpu-partitioning-2.9.0-1.el7.noarch
PASS
======================================================

:: [   LOG    ] :: === Test skew_stick in profile network-latency ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile cpu-partitioning ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-host ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-guest ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)

======================================================
Reproduced in:
    tuned-2.8.0-5.el7.noarch
    tuned-profiles-realtime-2.8.0-5.el7.noarch
    tuned-profiles-nfv-host-2.8.0-5.el7.noarch
    tuned-profiles-nfv-guest-2.8.0-5.el7.noarch
    tuned-profiles-cpu-partitioning-2.8.0-5.el7.noarch
FAIL
======================================================

:: [   LOG    ] :: === Test skew_stick in profile network-latency ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile cpu-partitioning ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-host ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-guest ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
Comment 28 errata-xmlrpc 2018-04-10 16:01:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0879
Note You need to log in before you can comment on or make changes to this bug.