Description of problem:
I can observe that the vCPU thread of the guest ran by run-tscdeadline-latency.sh being interrupted by QEMU's I/O thread. This increases the latency when trying to find the best timer advancement value from the host profile.
The problem is that all QEMU's threads are pinned to the same isolated host CPU in run-tscdeadline-latency.sh:
chrt -f 1 taskset -c $1 $QEMU [...]
We need to someone way move the vCPU thread to its own isolated CPU.
Version-Release number of selected component (if applicable): tuned-2.9.0-1.el7.noarch
I observed this issue on system that had many small issues affecting run-tscdeadline.sh fixed, I'm not sure it's going to be easy to observe this on an unfixed system where tscdeadline_latency.flat runs quick. But the procedure would be:
1. Pick an isolated CPU in the host (eg. CPU 1)
2. Change run-tscdeadline-latency.sh so that it runs only once (instead of going from 1000 to 7000 in 500 increments)
3. Start tracing for sched_switch on that CPU:
# trace-cmd record -esched_switch -M2
# ./run-tscdeadline-latency.sh 1
5. When run-tscdeadline-latency.sh finishes, run:
# trace-cmd list | grep qemu
When the issue triggers, you see two qemu PIDs preempting each other. If the problem doesn't show up, you can try it a few times or change tscdeadline_latency.flat to run for a longer duration (then I'm sure it will show up).
I don't know what's the best way to fix this. As a workaround, I wrote a python script that gets the vCPU thread id from QEMU via QMP and move it to another isolated core. But this seems overly complex, and QEMU has to be started with a QMP socket in paused mode (so that it doesn't start the measurement before the vCPU thread is moved).
Another two other ideas that may be much simpler:
1. Check if it's possible to tell if a qemu thread is a vCPU thread from /proc. This removes the need of having the Python script, but qemu still has to be started with -S
2. Carry a simple xml definition in the realtime-virtual-host profile and define the guest in libvirt on the spot, as libvirt supports everything we need
(In reply to Luiz Capitulino from comment #2)
> I don't know what's the best way to fix this. As a workaround, I wrote a
> python script that gets the vCPU thread id from QEMU via QMP and move it to
> another isolated core. But this seems overly complex, and QEMU has to be
> started with a QMP socket in paused mode (so that it doesn't start the
> measurement before the vCPU thread is moved).
> Another two other ideas that may be much simpler:
> 1. Check if it's possible to tell if a qemu thread is a vCPU thread from
> /proc. This removes the need of having the Python script, but qemu still has
> to be started with -S
Better use the official interface (QMP).
> 2. Carry a simple xml definition in the realtime-virtual-host profile and
> define the guest in libvirt on the spot, as libvirt supports everything we
Thats much more complex.
To me, your workaround is the proper fix (i would fix it the same way).
OK, I'll polish my script, integrate it in the profile and post it. I'll do this soon (ie. not right now).
Patch posted upstream, reassigning to Jaroslav
for the RHEL tuned integration.
*** Bug 1679007 has been marked as a duplicate of this bug. ***
*** Bug 1670275 has been marked as a duplicate of this bug. ***
We need this fix for 7.6.z.
In latest 7.6.z testing, below command can not return successfully, it's hang there. After applying patch of this bug, tuned can work well.
# tuned-adm profile realtime-virtual-host
As Luiz suggested, I add "7.6.z ?" flag here.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.