Bug 1554851

Summary: realtime-virtual-host: vCPU being interrupted by I/O thread
Product: Red Hat Enterprise Linux 7 Reporter: Luiz Capitulino <lcapitulino>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED ERRATA QA Contact: Robin Hack <rhack>
Severity: high Docs Contact:
Priority: urgent    
Version: 7.5CC: fkrska, jeder, jraju, jskarvad, lcapitulino, mtosatti, olysonek, pezhang, psklenar, rhack, snagar, thozza
Target Milestone: rcKeywords: Patch, Upstream, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tuned-2.11.0-0.1.rc1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1728699 (view as bug list) Environment:
Last Closed: 2019-08-06 13:04:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1240765, 1672377, 1728699    

Description Luiz Capitulino 2018-03-13 13:10:45 UTC
Description of problem:

I can observe that the vCPU thread of the guest ran by run-tscdeadline-latency.sh being interrupted by QEMU's I/O thread. This increases the latency when trying to find the best timer advancement value from the host profile.

The problem is that all QEMU's threads are pinned to the same isolated host CPU in run-tscdeadline-latency.sh:

    chrt -f 1 taskset -c $1 $QEMU [...]

We need to someone way move the vCPU thread to its own isolated CPU.

Version-Release number of selected component (if applicable):  tuned-2.9.0-1.el7.noarch


How reproducible:

I observed this issue on system that had many small issues affecting run-tscdeadline.sh fixed, I'm not sure it's going to be easy to observe this on an unfixed system where tscdeadline_latency.flat runs quick. But the procedure would be:

1. Pick an isolated CPU in the host (eg. CPU 1)

2. Change run-tscdeadline-latency.sh so that it runs only once (instead of going from 1000 to 7000 in 500 increments)

3. Start tracing for sched_switch on that CPU:

# trace-cmd record -esched_switch -M2

4. Run:

# ./run-tscdeadline-latency.sh 1

5. When run-tscdeadline-latency.sh finishes, run:

# trace-cmd list | grep qemu

When the issue triggers, you see two qemu PIDs preempting each other. If the problem doesn't show up, you can try it a few times or change tscdeadline_latency.flat to run for a longer duration (then I'm sure it will show up).

Comment 2 Luiz Capitulino 2018-03-13 13:19:08 UTC
I don't know what's the best way to fix this. As a workaround, I wrote a python script that gets the vCPU thread id from QEMU via QMP and move it to another isolated core. But this seems overly complex, and QEMU has to be started with a QMP socket in paused mode (so that it doesn't start the measurement before the vCPU thread is moved).

Another two other ideas that may be much simpler:

1. Check if it's possible to tell if a qemu thread is a vCPU thread from /proc. This removes the need of having the Python script, but qemu still has to be started with -S

2. Carry a simple xml definition in the realtime-virtual-host profile and define the guest in libvirt on the spot, as libvirt supports everything we need

Comment 3 Marcelo Tosatti 2018-03-14 21:06:49 UTC
(In reply to Luiz Capitulino from comment #2)
> I don't know what's the best way to fix this. As a workaround, I wrote a
> python script that gets the vCPU thread id from QEMU via QMP and move it to
> another isolated core. But this seems overly complex, and QEMU has to be
> started with a QMP socket in paused mode (so that it doesn't start the
> measurement before the vCPU thread is moved).
> 
> Another two other ideas that may be much simpler:
> 
> 1. Check if it's possible to tell if a qemu thread is a vCPU thread from
> /proc. This removes the need of having the Python script, but qemu still has
> to be started with -S

Better use the official interface (QMP). 

> 2. Carry a simple xml definition in the realtime-virtual-host profile and
> define the guest in libvirt on the spot, as libvirt supports everything we
> need

Thats much more complex.

To me, your workaround is the proper fix (i would fix it the same way).

Comment 4 Luiz Capitulino 2018-03-14 21:11:17 UTC
OK, I'll polish my script, integrate it in the profile and post it. I'll do this soon (ie. not right now).

Comment 5 Marcelo Tosatti 2018-07-05 15:34:53 UTC
Patch posted upstream, reassigning to Jaroslav 
for the RHEL tuned integration.

Comment 12 Jaison Raju 2019-02-28 05:25:11 UTC
*** Bug 1679007 has been marked as a duplicate of this bug. ***

Comment 13 Jaroslav Škarvada 2019-03-06 13:56:24 UTC
*** Bug 1670275 has been marked as a duplicate of this bug. ***

Comment 19 Pei Zhang 2019-07-08 03:17:56 UTC
We need this fix for 7.6.z. 

In latest 7.6.z testing, below command can not return successfully, it's hang there. After applying patch of this bug, tuned can work well.

# tuned-adm profile realtime-virtual-host


Versions:
3.10.0-957.29.1.rt56.941.el7.x86_64
tuned-2.10.0-6.el7_6.3.noarch

As Luiz suggested, I add "7.6.z ?" flag here.

Best regards,

Pei

Comment 25 errata-xmlrpc 2019-08-06 13:04:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2201