Very quickly tested with RHEL7's kernel-3.10.0-858.el7.x86_64 and the cpu-partitioning profile, can't reproduce. Meaning, I don't see any preemptions at all. However, I ran the test-case only for a few minutes.

Let me emphasize that this issue is extremely important. We have a confirmed 20% worse latency for 10 minutes runs and I'd expect that right now RHEL+cpu-partitioning offers better latency for DPDK than the RT kernel.

Also, for regular RT usage, if you have the tick ticking at 1000 times per second you'll have 1000 preemptions per second.

Here's a reproducer that doesn't require KVM-RT profiles, but still requires you to completely isolate a CPU:

1. Completely isolate a CPU, use nohz_full, etc
2. Run a hog application pinned to that CPU: taskset -c CPU ./hog
3. Trace sched_switch events

On an unfixed kernel, there will be one context switch to the ktimersoftd thread per second. On a fixed system, there should be no sched_switch events whatsoever.

Hi Luis,

I am already reproducing the issue in a local system. Currently checking upstream code/reproducer.

I have a question: It is not related to this BZ, but I saw rcuc threads being awakened. To avoid this, we can use rcu_nocb_poll, but I do not see this being enabled in the realtime-virtual-host. Is there a reason for it not being enabled?

-- Daniel

Can you open a BZ? You can assign it to me.

Because of another issue I'm debugging, I've just traced sched_switch on the host when running KVM-RT test-case and didn't see any context switches to rcuc threads. But in any case it's better to investigate.

Here is a resume of my findings so far.

I could not reproduce the rcuc wakeup anymore. I do not know why. So I will not file the BZ until I see this again, if I see... maybe there was something wrong with my setup.

Using the kernel with the new timer wheel, I still saw the wakeup of the ktimersoftd. But after applying the patches:

[PATCH v4 1/2] timers: Don't wake ktimersoftd on every tick
[PATCH v4 2/2] timers: Don't search for expired timers while TIMER_SOFTIRQ is scheduled

And adding the following kernel command line parameter:

tsc=reliable

to avoid the timer that checks the stability of the TSC.

The above-mentioned patches are not part of the PREEMPT_RT patch set yet, so there might be newer versions.

I can see very few ktimersoftd wake-up. Those that I see seems to be legit, and rarely take place, and each one after a long time one from the previous one.

My next step is to try to backport the timer wheel for the RHEL-7.6 code.

Thanks
-- Daniel

Daniel,

Does the first patch depend on the new timer wheel?

Hi Luis,

Unfortunately, yes. These patches improve the timer wheel by forwarding it in the interrupt context. The softirq is then raised just in case of an armed timer in the forwarding process.

-- Daniel

The dependencies for the timer wheel update, first half of the solution for the problem described in this bugzilla ticket, have already been added to kernel-rt-3.10.0-900.rt56.846.el7. These changes are enough to keep us in sync with the changes added to RHEL, respecting the kernel-rt differences.

Changes added:

bb2b1db2d575 timers: Reduce the CPU index space to 256k
34e2660305c9 timers: Use proper base migration in add_timer_on()
9a9ececb8d90 hlist: Add hlist_is_singular_node() helper
0fbdb2309b1a signals: Use hrtimer for sigtimedwait()
f68a3f9917ed timers: Remove the deprecated mod_timer_pinned() API
4da22cacb04c timers, driver/net/ethernet/tile: Initialize the egress timer as pinned
b1aa9f139e6b timers, cpufreq/powernv: Initialize the gpstate timer as pinned
c274cd526b70 timers, x86/apic/uv: Initialize the UV heartbeat timer as pinned
dee6b36a6cc9 timers: Make 'pinned' a timer property
da4f00fe9a87 timer: Minimize nohz off overhead
e79103755f45 timer: Reduce timer migration overhead if disabled (v2)
417bd5c3fdc1 Remove code redundancy while calling get_nohz_timer_target()
dde8ca171b2a timer: Stats: Simplify the flags handling
ce38ad8067ac timer: Replace timer base by a cpu index
6e878157ddce timer: Use timer->base for flag checks
563b84b3a535 tracing: timer: Add deferrable flag to timer_start
66743e988f03 timer: Use hlist for the timer wheel hash buckets
45665e16b954 timer: Remove FIFO "guarantee"
6056d7fc3054 timers: Sanitize catchup_timer_jiffies() usage
9b05865436d7 timer: Put usleep_range into the __sched section
2e80b77a26ca timer: Remove pointless return value of do_usleep_range()
c57ba680b598 timer: Further simplify the SMP and HOTPLUG logic
9960cc117fd1 timer: Don't initialize 'tvec_base' on hotplug
c0327baf7493 timer: Allocate per-cpu tvec_base's statically

Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.

Hi,

We have the main dependencies for this patch set already in our kernel. But, the problem is still present in the upstream kernel.
We are working with the kernel maintainer (Timer/RT) to find a solution, and this is a high priority BZ for us.
However, this is a very complex problem and will require some time to find an upstream solution.

Thanks!
-- Daniel

doc text reviewed, It is fine.

Thanks!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1070