Bug 1451073

Summary:	tuned: use skew_tick=1 in all low-latency profiles
Product:	Red Hat Enterprise Linux 7	Reporter:	Luiz Capitulino <lcapitulino>
Component:	tuned	Assignee:	Jaroslav Škarvada <jskarvad>
Status:	CLOSED ERRATA	QA Contact:	Tereza Cerna <tcerna>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.4	CC:	jeder, jmario, jskarvad, lpol, olysonek, psklenar, salmy, tcerna, thozza
Target Milestone:	rc	Keywords:	Patch, Upstream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tuned-2.9.0-0.1.rc1.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-10 16:01:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1465887, 1467576, 1485946

Description Luiz Capitulino 2017-05-15 17:11:50 UTC

Description of problem:

On Linux by default the tick timer is programmed to fire at the same time on all CPUs. This may cause lock contention as some locks are taken by the tick handler, as the jiffies_lock and an RCU lock.

The skew_tick=1 parameter causes the kernel to program each CPU's tick timer to fire at different times, avoiding any possible lock contention. Kernel developers advise passing skew_tick=1 to the kernel when the workload is very sensible to jittier.

Due to this reason, we're considering using skew_tick=1 in the network-latency profile, which will cause it to be inherited by the cpu-partitioning profile and realtime profiles (although the realtime profiles are already getting skew_tick support see bug 1447938).

This BZ should gather data in favour or against using skew_tick=1.


Version-Release number of selected component (if applicable): tuned-2.8.0-3.el7.noarch

Comment 2 Luiz Capitulino 2017-05-15 17:18:42 UTC

I'll do the investigatin and post the patches in case we decide to use skew_tick=1, so I'm taking the BZ.

Comment 3 Joe Mario 2017-05-16 13:47:12 UTC

A few thoughts:

 - Current thinking is skew_tick=1 is only relevant when nohz_full is NOT used.
   Though it should not hurt if used with nohz_full.

 - Do we know of any downside to skewing the timer ticks?  I know of none, 
   but since this is not a widely used flag, it's worth understanding.

Joe

Comment 4 Luiz Capitulino 2017-05-16 18:33:03 UTC

The only known downside is that skew_tick=1 could theoretically cause higher power consumption because a socket would take more time (or be unable) to shut itself down when idle. However, this hasn't been measured on recent kernels.

Regarding nohz_full, I guess you may be referring to bug 1404440. In this bug, sysjitter behavior with nohz_full cause so much jitter that skew_tick=1 is probably not going to make much difference. However, it's not clear if this behavior is specific to sysjitter. This means that there might be cases where skew_tick=1 works well with nohz_full= (I expect this to be the case with RT for example, although I haven't checked that yet).

Comment 5 Joe Mario 2017-05-16 18:44:44 UTC

Hi Luiz:
 Your reply made me realize the error in my thinking for my bullet 1 above.

 I was thinking nohz_full brings the ticks down to 1/sec, which negates the risk of the tick thundering herd problem.

 But if there is more than one thread on the cpu, then there will be 1000 ticks/sec (which I forgot in my comment-3 reply.

 So yes do include skew_tick=1 when nohz_full is used.

Joe

Comment 6 Jaroslav Škarvada 2017-05-29 15:51:51 UTC

Upstream commit fixing the problem:
https://github.com/redhat-performance/tuned/commit/9bc020df37d49b065aac0b164f0cc3ec254561a5

Comment 7 Jaroslav Škarvada 2017-06-02 09:18:20 UTC

(In reply to Jaroslav Škarvada from comment #6)
> Upstream commit fixing the problem:
> https://github.com/redhat-performance/tuned/commit/
> 9bc020df37d49b065aac0b164f0cc3ec254561a5

The mentioned upstream commit is for RT profiles only.

Comment 8 Luiz Capitulino 2017-06-02 13:18:13 UTC

Yes, this one is not a blocker and is not being fixed yet. Moving to 7.5.

Comment 10 Jaroslav Škarvada 2017-07-19 11:55:42 UTC

Fixed by the following upstream commit:
https://github.com/redhat-performance/tuned/commit/643864f10cbf2891d8fd2b4c33ec358904ff209b

Comment 11 Luiz Capitulino 2017-07-19 12:56:12 UTC

Jaroslav,

The cpu-partitioning profile defines "cmdline" in the bootloader section. Shouldn't you change it as you did in commit 4483641602? Btw, I didn't check if the other profiles including network-latency define cmdline too.

Comment 12 Jaroslav Škarvada 2017-07-19 13:08:44 UTC

(In reply to Luiz Capitulino from comment #11)
> Jaroslav,
> 
> The cpu-partitioning profile defines "cmdline" in the bootloader section.
> Shouldn't you change it as you did in commit 4483641602? Btw, I didn't check
> if the other profiles including network-latency define cmdline too.

Thanks, it would miss skew_tick=1 in cpu_partitioning. Fixed by:
https://github.com/redhat-performance/tuned/commit/d1379afbec81a24e3c3c8444654cd6e00f0eb96d

AFAIK there are no more profiles including it.

Comment 13 Luiz Capitulino 2017-07-19 13:10:24 UTC

Excellent, thanks!

Comment 14 Jaroslav Škarvada 2017-07-19 13:11:47 UTC

(In reply to Jaroslav Škarvada from comment #12)
> AFAIK there are no more profiles including it.

Not counting the realtime which I already fixed in the previous commit.

Comment 15 Luiz Capitulino 2017-10-03 20:51:30 UTC

The cpu-partitioning profile needs skew_tick=1 for 7.4.z. If this change is too complex to backport, we can just set "skew_tick=1" in the bootloader section for cpu-partitioning.

Comment 18 Fedora Update System 2017-10-13 14:20:46 UTC

tuned-2.9.0-0.1.rc1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df

Comment 20 Fedora Update System 2017-10-13 22:25:11 UTC

tuned-2.9.0-0.1.rc1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df

Comment 21 Fedora Update System 2017-10-13 23:24:59 UTC

tuned-2.9.0-0.1.rc1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5f0849d207

Comment 22 Fedora Update System 2017-10-29 21:07:00 UTC

tuned-2.9.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0e45ce4685

Comment 23 Fedora Update System 2017-10-29 21:13:07 UTC

tuned-2.9.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c30e9bd1ea

Comment 24 Luiz Capitulino 2018-01-22 22:02:10 UTC

I verified this BZ. I checked that skew_tick=1 option is passed to the kernel when using profiles: cpu-partitioning, network-latency, realtime, realtime-virtual-host and realtime-virtual-guest.

Comment 25 Tereza Cerna 2018-01-26 12:39:00 UTC

New test case for this bugzilla was created:
/CoreOS/tuned/Regression/bz1451073-use_skew_tick_in_cmdline

======================================================
Verified in:
    tuned-2.9.0-1.el7.noarch
    tuned-profiles-realtime-2.9.0-1.el7.noarch
    tuned-profiles-nfv-host-2.9.0-1.el7.noarch
    tuned-profiles-nfv-guest-2.9.0-1.el7.noarch
    tuned-profiles-cpu-partitioning-2.9.0-1.el7.noarch
PASS
======================================================

:: [   LOG    ] :: === Test skew_stick in profile network-latency ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile cpu-partitioning ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-host ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-guest ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)

======================================================
Reproduced in:
    tuned-2.8.0-5.el7.noarch
    tuned-profiles-realtime-2.8.0-5.el7.noarch
    tuned-profiles-nfv-host-2.8.0-5.el7.noarch
    tuned-profiles-nfv-guest-2.8.0-5.el7.noarch
    tuned-profiles-cpu-partitioning-2.8.0-5.el7.noarch
FAIL
======================================================

:: [   LOG    ] :: === Test skew_stick in profile network-latency ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile cpu-partitioning ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-host ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)
:: [   LOG    ] :: === Test skew_stick in profile realtime-virtual-guest ===
:: [   PASS   ] :: Command 'tuned-adm profile network-latency' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep skew_tick=1' (Expected 0, got 1)
:: [   LOG    ] :: === Set to default profile ===
:: [   PASS   ] :: Command 'tuned-adm profile virtual-host' (Expected 0, got 0)
:: [   PASS   ] :: Command 'grub2-mkconfig' (Expected 0, got 0)
:: [   PASS   ] :: Command 'cat /boot/grub2/grub.cfg | grep tuned_params= | grep -v skew_tick=1' (Expected 0, got 0)

Comment 28 errata-xmlrpc 2018-04-10 16:01:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0879