RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1757165 - 8 vCPU guest need max latency < 20 us with stress [RT-8.2]
Summary: 8 vCPU guest need max latency < 20 us with stress [RT-8.2]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: kernel-rt
Version: 8.2
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: rc
: 8.2
Assignee: Marcelo Tosatti
QA Contact: Pei Zhang
URL:
Whiteboard:
: 1761735 (view as bug list)
Depends On:
Blocks: 1640832 1680412 1722609 1813007
TreeView+ depends on / blocked
 
Reported: 2019-09-30 17:45 UTC by Beth Uptagrafft
Modified: 2022-03-08 10:33 UTC (History)
25 users (show)

Fixed In Version: kernel-rt-4.18.0-148.rt13.5.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1690543
: 1813007 (view as bug list)
Environment:
Last Closed: 2020-04-28 15:25:32 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pvaanane: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-28268 0 None None None 2022-03-08 10:32:50 UTC
Red Hat Issue Tracker RHELPLAN-38805 0 None None None 2022-03-08 10:33:03 UTC
Red Hat Product Errata RHSA-2020:1567 0 None None None 2020-04-28 15:27:05 UTC

Comment 5 Marcelo Tosatti 2019-11-12 12:37:47 UTC 
Pei, do you have the "nowatchdog" kernel command line option set, on both host and guest?
Comment 6 Pei Zhang 2019-11-12 12:44:58 UTC 
(In reply to Marcelo Tosatti from comment #5)
> Pei, do you have the "nowatchdog" kernel command line option set, on both
> host and guest?

Hi Marcelo,

Yes, nowatchdog is in both host and guest kernel line.

Best regards,

Pei
Comment 7 Marcelo Tosatti 2019-11-12 19:27:56 UTC 
(In reply to Pei Zhang from comment #6)
> (In reply to Marcelo Tosatti from comment #5)
> > Pei, do you have the "nowatchdog" kernel command line option set, on both
> > host and guest?
> 
> Hi Marcelo,
> 
> Yes, nowatchdog is in both host and guest kernel line.
> 
> Best regards,
> 
> Pei

Hi Pei,

Can you please provide access to the machine?

My tests show max=17us.

TIA
Comment 20 Pei Zhang 2020-02-28 15:56:41 UTC 
Verified with kernel-rt-4.18.0-184.rt13.42.el8.x86_64:

KVM-RT testing get PASS:  24h cyclictest max latency is 21us.

==Results==
(1)Single VM with 1 rt vCPU:
# Min Latencies: 00004
# Avg Latencies: 00005
# Max Latencies: 00018

(2)Single VM with 8 rt vCPUs:
# Min Latencies: 00004 00004 00004 00005 00004 00005 00005 00005
# Avg Latencies: 00005 00005 00005 00005 00005 00005 00005 00005
# Max Latencies: 00017 00014 00015 00015 00016 00019 00016 00014

(3)Multiple VMs each with 1 rt vCPU:
- VM1
# Min Latencies: 00004
# Avg Latencies: 00004
# Max Latencies: 00021

- VM2
# Min Latencies: 00004
# Avg Latencies: 00005
# Max Latencies: 00017

- VM3
# Min Latencies: 00004
# Avg Latencies: 00004
# Max Latencies: 00016

- VM4
# Min Latencies: 00004
# Avg Latencies: 00004
# Max Latencies: 00017


==Versions==
kernel-rt-4.18.0-184.rt13.42.el8.x86_64
tuned-2.13.0-5.el8.noarch
microcode_ctl-20191115-4.el8.x86_64
rt-tests-1.5-18.el8.x86_64
qemu-kvm-4.2.0-12.module+el8.2.0+5858+afd073bc.x86_64
python3-libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64

So this bug has been fixed very well. Move to 'VERIFIED'.
Comment 25 Luiz Capitulino 2020-03-02 02:23:21 UTC 
(In reply to Pei Zhang from comment #20)
> Verified with kernel-rt-4.18.0-184.rt13.42.el8.x86_64:
> 
> KVM-RT testing get PASS:  24h cyclictest max latency is 21us.
> 
> ==Results==
> (1)Single VM with 1 rt vCPU:
> # Min Latencies: 00004
> # Avg Latencies: 00005
> # Max Latencies: 00018
> 
> (2)Single VM with 8 rt vCPUs:
> # Min Latencies: 00004 00004 00004 00005 00004 00005 00005 00005
> # Avg Latencies: 00005 00005 00005 00005 00005 00005 00005 00005
> # Max Latencies: 00017 00014 00015 00015 00016 00019 00016 00014
> 
> (3)Multiple VMs each with 1 rt vCPU:
> - VM1
> # Min Latencies: 00004
> # Avg Latencies: 00004
> # Max Latencies: 00021

Pei, since we get max latency greater than 20us in
this case, maybe this BZ has failed verification?

I mean, for KVM-RT automated testing, having a max
latency of about 20us is good enough. However, this
BZ is about achieving less than 20us.

> 
> - VM2
> # Min Latencies: 00004
> # Avg Latencies: 00005
> # Max Latencies: 00017
> 
> - VM3
> # Min Latencies: 00004
> # Avg Latencies: 00004
> # Max Latencies: 00016
> 
> - VM4
> # Min Latencies: 00004
> # Avg Latencies: 00004
> # Max Latencies: 00017
> 
> 
> ==Versions==
> kernel-rt-4.18.0-184.rt13.42.el8.x86_64
> tuned-2.13.0-5.el8.noarch
> microcode_ctl-20191115-4.el8.x86_64
> rt-tests-1.5-18.el8.x86_64
> qemu-kvm-4.2.0-12.module+el8.2.0+5858+afd073bc.x86_64
> python3-libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64
> 
> So this bug has been fixed very well. Move to 'VERIFIED'.
Comment 26 Pei Zhang 2020-03-02 03:01:45 UTC 
(In reply to Luiz Capitulino from comment #25)
...
> > 
> > (3)Multiple VMs each with 1 rt vCPU:
> > - VM1
> > # Min Latencies: 00004
> > # Avg Latencies: 00004
> > # Max Latencies: 00021
> 
> Pei, since we get max latency greater than 20us in
> this case, maybe this BZ has failed verification?
> 
> I mean, for KVM-RT automated testing, having a max
> latency of about 20us is good enough. However, this
> BZ is about achieving less than 20us.
> 

Hi Luiz,

Make sense. I agree we should have strict latency results to verify this bug.

This run was testing with default /sys/kernel/ktimer_lockless_check 0.  I'll try with /sys/kernel/ktimer_lockless_check 1 to see if the testing results can achieve <=20us. Testing results will be updated soon.

I think I need to recover to ON_QA status now. 

Thank you.

Best regards,

Pei
Comment 27 Pei Zhang 2020-03-05 01:08:21 UTC 
With "# echo 1 > /sys/kernel/ktimer_lockless_check", the slight spike (1us) is gone.

[PASS]With mitigation off, 24h cyclictest max latency is 19us.

Hightlight:
1. This run was testing with "# echo 1 > /sys/kernel/ktimer_lockless_check" in both host and guest.

2. This run was testing with manually adding "isolcpus=managed_irq," in /boot/grub2/grub.cfg in both host and guest.


==Results==
(1)Single VM with 1 rt vCPU:
# Min Latencies: 00005
# Avg Latencies: 00006
# Max Latencies: 00019

(2)Single VM with 8 rt vCPUs:
# Min Latencies: 00005 00007 00007 00007 00007 00007 00007 00007
# Avg Latencies: 00005 00007 00008 00007 00007 00007 00007 00007
# Max Latencies: 00017 00015 00018 00014 00016 00015 00015 00015

(3)Multiple VMs each with 1 rt vCPU:
- VM1
# Min Latencies: 00004
# Avg Latencies: 00005
# Max Latencies: 00015

- VM2
# Min Latencies: 00004
# Avg Latencies: 00005
# Max Latencies: 00015

- VM3
# Min Latencies: 00004
# Avg Latencies: 00005
# Max Latencies: 00015

- VM4
# Min Latencies: 00004
# Avg Latencies: 00005
# Max Latencies: 00019

==Versions==
kernel-rt-4.18.0-185.rt13.43.el8.x86_64
tuned-2.13.0-5.el8.noarch
microcode_ctl-20191115-4.el8.x86_64
rt-tests-1.5-18.el8.x86_64
qemu-kvm-4.2.0-12.module+el8.2.0+5858+afd073bc.x86_64
python3-libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64
Comment 29 Pei Zhang 2020-03-06 01:23:55 UTC 
As Comment 27 and Comment 28,  24h cyclictest max latency both < 20us (2/2 PASS). 

Move this bug to 'VERIFIED'.
Comment 32 Luiz Capitulino 2020-03-09 18:35:50 UTC 
*** Bug 1761735 has been marked as a duplicate of this bug. ***
Comment 43 errata-xmlrpc 2020-04-28 15:25:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1567
Note You need to log in before you can comment on or make changes to this bug.