Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionLuiz Capitulino
2018-03-13 13:10:45 UTC
Description of problem:
I can observe that the vCPU thread of the guest ran by run-tscdeadline-latency.sh being interrupted by QEMU's I/O thread. This increases the latency when trying to find the best timer advancement value from the host profile.
The problem is that all QEMU's threads are pinned to the same isolated host CPU in run-tscdeadline-latency.sh:
chrt -f 1 taskset -c $1 $QEMU [...]
We need to someone way move the vCPU thread to its own isolated CPU.
Version-Release number of selected component (if applicable): tuned-2.9.0-1.el7.noarch
How reproducible:
I observed this issue on system that had many small issues affecting run-tscdeadline.sh fixed, I'm not sure it's going to be easy to observe this on an unfixed system where tscdeadline_latency.flat runs quick. But the procedure would be:
1. Pick an isolated CPU in the host (eg. CPU 1)
2. Change run-tscdeadline-latency.sh so that it runs only once (instead of going from 1000 to 7000 in 500 increments)
3. Start tracing for sched_switch on that CPU:
# trace-cmd record -esched_switch -M2
4. Run:
# ./run-tscdeadline-latency.sh 1
5. When run-tscdeadline-latency.sh finishes, run:
# trace-cmd list | grep qemu
When the issue triggers, you see two qemu PIDs preempting each other. If the problem doesn't show up, you can try it a few times or change tscdeadline_latency.flat to run for a longer duration (then I'm sure it will show up).
I don't know what's the best way to fix this. As a workaround, I wrote a python script that gets the vCPU thread id from QEMU via QMP and move it to another isolated core. But this seems overly complex, and QEMU has to be started with a QMP socket in paused mode (so that it doesn't start the measurement before the vCPU thread is moved).
Another two other ideas that may be much simpler:
1. Check if it's possible to tell if a qemu thread is a vCPU thread from /proc. This removes the need of having the Python script, but qemu still has to be started with -S
2. Carry a simple xml definition in the realtime-virtual-host profile and define the guest in libvirt on the spot, as libvirt supports everything we need
(In reply to Luiz Capitulino from comment #2)
> I don't know what's the best way to fix this. As a workaround, I wrote a
> python script that gets the vCPU thread id from QEMU via QMP and move it to
> another isolated core. But this seems overly complex, and QEMU has to be
> started with a QMP socket in paused mode (so that it doesn't start the
> measurement before the vCPU thread is moved).
>
> Another two other ideas that may be much simpler:
>
> 1. Check if it's possible to tell if a qemu thread is a vCPU thread from
> /proc. This removes the need of having the Python script, but qemu still has
> to be started with -S
Better use the official interface (QMP).
> 2. Carry a simple xml definition in the realtime-virtual-host profile and
> define the guest in libvirt on the spot, as libvirt supports everything we
> need
Thats much more complex.
To me, your workaround is the proper fix (i would fix it the same way).
We need this fix for 7.6.z.
In latest 7.6.z testing, below command can not return successfully, it's hang there. After applying patch of this bug, tuned can work well.
# tuned-adm profile realtime-virtual-host
Versions:
3.10.0-957.29.1.rt56.941.el7.x86_64
tuned-2.10.0-6.el7_6.3.noarch
As Luiz suggested, I add "7.6.z ?" flag here.
Best regards,
Pei
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2019:2201
Description of problem: I can observe that the vCPU thread of the guest ran by run-tscdeadline-latency.sh being interrupted by QEMU's I/O thread. This increases the latency when trying to find the best timer advancement value from the host profile. The problem is that all QEMU's threads are pinned to the same isolated host CPU in run-tscdeadline-latency.sh: chrt -f 1 taskset -c $1 $QEMU [...] We need to someone way move the vCPU thread to its own isolated CPU. Version-Release number of selected component (if applicable): tuned-2.9.0-1.el7.noarch How reproducible: I observed this issue on system that had many small issues affecting run-tscdeadline.sh fixed, I'm not sure it's going to be easy to observe this on an unfixed system where tscdeadline_latency.flat runs quick. But the procedure would be: 1. Pick an isolated CPU in the host (eg. CPU 1) 2. Change run-tscdeadline-latency.sh so that it runs only once (instead of going from 1000 to 7000 in 500 increments) 3. Start tracing for sched_switch on that CPU: # trace-cmd record -esched_switch -M2 4. Run: # ./run-tscdeadline-latency.sh 1 5. When run-tscdeadline-latency.sh finishes, run: # trace-cmd list | grep qemu When the issue triggers, you see two qemu PIDs preempting each other. If the problem doesn't show up, you can try it a few times or change tscdeadline_latency.flat to run for a longer duration (then I'm sure it will show up).