Bug 1451073 - tuned: use skew_tick=1 in all low-latency profiles
tuned: use skew_tick=1 in all low-latency profiles
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned (Show other bugs)
7.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Jaroslav Škarvada
Tereza Cerna
: Patch, Upstream
Depends On:
Blocks: 1485946 1465887 TUNED-7.5-REBASE
  Show dependency treegraph
 
Reported: 2017-05-15 13:11 EDT by Luiz Capitulino
Modified: 2018-04-10 12:01 EDT (History)
9 users (show)

See Also:
Fixed In Version: tuned-2.9.0-0.1.rc1.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-10 12:01:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Luiz Capitulino 2017-05-15 13:11:50 EDT
Description of problem:

On Linux by default the tick timer is programmed to fire at the same time on all CPUs. This may cause lock contention as some locks are taken by the tick handler, as the jiffies_lock and an RCU lock.

The skew_tick=1 parameter causes the kernel to program each CPU's tick timer to fire at different times, avoiding any possible lock contention. Kernel developers advise passing skew_tick=1 to the kernel when the workload is very sensible to jittier.

Due to this reason, we're considering using skew_tick=1 in the network-latency profile, which will cause it to be inherited by the cpu-partitioning profile and realtime profiles (although the realtime profiles are already getting skew_tick support see bug 1447938).

This BZ should gather data in favour or against using skew_tick=1.


Version-Release number of selected component (if applicable): tuned-2.8.0-3.el7.noarch
Comment 2 Luiz Capitulino 2017-05-15 13:18:42 EDT
I'll do the investigatin and post the patches in case we decide to use skew_tick=1, so I'm taking the BZ.
Comment 3 Joe Mario 2017-05-16 09:47:12 EDT
A few thoughts:

 - Current thinking is skew_tick=1 is only relevant when nohz_full is NOT used.
   Though it should not hurt if used with nohz_full.

 - Do we know of any downside to skewing the timer ticks?  I know of none, 
   but since this is not a widely used flag, it's worth understanding.

Joe
Comment 4 Luiz Capitulino 2017-05-16 14:33:03 EDT
The only known downside is that skew_tick=1 could theoretically cause higher power consumption because a socket would take more time (or be unable) to shut itself down when idle. However, this hasn't been measured on recent kernels.

Regarding nohz_full, I guess you may be referring to bug 1404440. In this bug, sysjitter behavior with nohz_full cause so much jitter that skew_tick=1 is probably not going to make much difference. However, it's not clear if this behavior is specific to sysjitter. This means that there might be cases where skew_tick=1 works well with nohz_full= (I expect this to be the case with RT for example, although I haven't checked that yet).
Comment 5 Joe Mario 2017-05-16 14:44:44 EDT
Hi Luiz:
 Your reply made me realize the error in my thinking for my bullet 1 above.

 I was thinking nohz_full brings the ticks down to 1/sec, which negates the risk of the tick thundering herd problem.

 But if there is more than one thread on the cpu, then there will be 1000 ticks/sec (which I forgot in my comment-3 reply.

 So yes do include skew_tick=1 when nohz_full is used.

Joe
Comment 6 Jaroslav Škarvada 2017-05-29 11:51:51 EDT
Upstream commit fixing the problem:
https://github.com/redhat-performance/tuned/commit/9bc020df37d49b065aac0b164f0cc3ec254561a5
Comment 7 Jaroslav Škarvada 2017-06-02 05:18:20 EDT
(In reply to Jaroslav Škarvada from comment #6)
> Upstream commit fixing the problem:
> https://github.com/redhat-performance/tuned/commit/
> 9bc020df37d49b065aac0b164f0cc3ec254561a5

The mentioned upstream commit is for RT profiles only.
Comment 8 Luiz Capitulino 2017-06-02 09:18:13 EDT
Yes, this one is not a blocker and is not being fixed yet. Moving to 7.5.
Comment 10 Jaroslav Škarvada 2017-07-19 07:55:42 EDT
Fixed by the following upstream commit:
https://github.com/redhat-performance/tuned/commit/643864f10cbf2891d8fd2b4c33ec358904ff209b
Comment 11 Luiz Capitulino 2017-07-19 08:56:12 EDT
Jaroslav,

The cpu-partitioning profile defines "cmdline" in the bootloader section. Shouldn't you change it as you did in commit 4483641602? Btw, I didn't check if the other profiles including network-latency define cmdline too.
Comment 12 Jaroslav Škarvada 2017-07-19 09:08:44 EDT
(In reply to Luiz Capitulino from comment #11)
> Jaroslav,
> 
> The cpu-partitioning profile defines "cmdline" in the bootloader section.
> Shouldn't you change it as you did in commit 4483641602? Btw, I didn't check
> if the other profiles including network-latency define cmdline too.

Thanks, it would miss skew_tick=1 in cpu_partitioning. Fixed by:
https://github.com/redhat-performance/tuned/commit/d1379afbec81a24e3c3c8444654cd6e00f0eb96d

AFAIK there are no more profiles including it.
Comment 13 Luiz Capitulino 2017-07-19 09:10:24 EDT
Excellent, thanks!
Comment 14 Jaroslav Škarvada 2017-07-19 09:11:47 EDT
(In reply to Jaroslav Škarvada from comment #12)
> AFAIK there are no more profiles including it.

Not counting the realtime which I already fixed in the previous commit.
Comment 15 Luiz Capitulino 2017-10-03 16:51:30 EDT
The cpu-partitioning profile needs skew_tick=1 for 7.4.z. If this change is too complex to backport, we can just set "skew_tick=1" in the bootloader section for cpu-partitioning.
Comment 18 Fedora Update System 2017-10-13 10:20:46 EDT
tuned-2.9.0-0.1.rc1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df
Comment 20 Fedora Update System 2017-10-13 18:25:11 EDT
tuned-2.9.0-0.1.rc1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df
Comment 21 Fedora Update System 2017-10-13 19:24:59 EDT
tuned-2.9.0-0.1.rc1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5f0849d207
Comment 22 Fedora Update System 2017-10-29 17:07:00 EDT
tuned-2.9.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0e45ce4685
Comment 23 Fedora Update System 2017-10-29 17:13:07 EDT
tuned-2.9.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c30e9bd1ea
Comment 24 Luiz Capitulino 2018-01-22 17:02:10 EST
I verified this BZ. I checked that skew_tick=1 option is passed to the kernel when using profiles: cpu-partitioning, network-latency, realtime, realtime-virtual-host and realtime-virtual-guest.
Comment 25 Tereza Cerna 2018-01-26 07:39:00 EST
New test case for this bugzilla was created:
/CoreOS/tuned/Regression/bz1451073-use_skew_tick_in_cmdline

======================================================
Verified in:
    tuned-2.9.0-1.el7.noarch
    tuned-profiles-realtime-2.9.0-1.el7.noarch
    tuned-profiles-nfv-host-2.9.0-1.el7.noarch
    tuned-profiles-nfv-guest-2.9.0-1.el7.noarch
    tuned-profiles-cpu-partitioning-2.9.0-1.el7.noarch
PASS
======================================================

:: [   LOG    ] :: === Test skew_stick in profile network-latency ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile cpu-partitioning ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-host ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-guest ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)

======================================================
Reproduced in:
    tuned-2.8.0-5.el7.noarch
    tuned-profiles-realtime-2.8.0-5.el7.noarch
    tuned-profiles-nfv-host-2.8.0-5.el7.noarch
    tuned-profiles-nfv-guest-2.8.0-5.el7.noarch
    tuned-profiles-cpu-partitioning-2.8.0-5.el7.noarch
FAIL
======================================================

:: [   LOG    ] :: === Test skew_stick in profile network-latency ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile cpu-partitioning ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-host ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-guest ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
Comment 28 errata-xmlrpc 2018-04-10 12:01:26 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0879

Note You need to log in before you can comment on or make changes to this bug.