Bug 1761735
| Summary: | cyclictest with default interval exceeds 20us maximum | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Pei Zhang <pezhang> |
| Component: | kernel-rt | Assignee: | Luiz Capitulino <lcapitulino> |
| kernel-rt sub component: | KVM | QA Contact: | Pei Zhang <pezhang> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | bhu, chayang, jinzhao, jkacur, jlelli, juzhang, mtosatti, virt-maint, williams |
| Version: | 8.2 | Flags: | pm-rhel:
mirror+
|
| Target Milestone: | rc | ||
| Target Release: | 8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-09 18:35:50 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1640832, 1680412 | ||
Hi, maybe dumb question, however, (In reply to Pei Zhang from comment #0) [...] > Version-Release number of selected component (if applicable): > > kernel-rt-4.18.0-147.rt24.93.el8.x86_64 > tuned-2.12.0-3.el8.noarch > qemu-kvm-4.1.0-13.module+el8.1.0+4313+ef76ec61.x86_64 > libvirt-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64 What version of rt-tests is in use? > 5. Start cyclictest in guest with default interval value. > > # taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h > -h60 -t 1 -a 1 -b40 --notrace Or maybe this is compiled from source (looking at path). In any case, doesn't "-b40" option cause cyclictest to exit early if 40us threshold is crossed? (In reply to Juri Lelli from comment #1) [...] > > What version of rt-tests is in use? Hi Juri, I'm using cyclictest tool from rt-tests-1.0-12.el7. The cyclictest tool is saved and come from fix path in my testing setup: /home/nfv-virt-rt-kvm/tools/cyclictest. In order to compare result between rhel7 and rhel8, I chose using exactly same tool. This is because at the beginning testing phase of rhel8.0.0, it's cyclictest (from rt-tests) didn't work well, so we decided to use tool from rhel7 and I just copy it from rt-tests-1.0-12.el7(from March 1, 2019 around) and keep using this version in all my next testings (even now)on both rhel7 and rhel8. (Next, I may switch to cyclictest from latest rt-tests) > > > 5. Start cyclictest in guest with default interval value. > > > > # taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h > > -h60 -t 1 -a 1 -b40 --notrace > > Or maybe this is compiled from source (looking at path). > > In any case, doesn't "-b40" option cause cyclictest to exit early if > 40us threshold is crossed? Sorry, I made a typo here. The full command line should be(no -b40): # taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 1 -a 1 --notrace I'd like to update the testing details here (with default interval): (1)Single VM with 1 rt vCPU: Test started at: 2019-09-30 10:32:13 Monday Kernel cmdline: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 root=/dev/mapper/rhel_vm--74--33-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm--74--33-swap rd.lvm.lv=rhel_vm-74-33/root rd.lvm.lv=rhel_vm-74-33/swap skew_tick=1 isolcpus=1 intel_pstate=disable nosoftlockup nohz=on nohz_full=1 rcu_nocbs=1 default_hugepagesz=1G iommu=pt intel_iommu=on kvm-intel.vmentry_l1d_flush=never X86 debug pts: pti_enable= ibpb_enabled= ibrs_enabled= retp_enabled= Machine: vm-74-33.lab.eng.pek2.redhat.com CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Test duration(plan): 24h Test ended at: 2019-10-01 10:32:20 Tuesday cyclictest cmdline: taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 1 -a 1 --notrace cyclictest results: # Min Latencies: 00007 # Avg Latencies: 00010 # Max Latencies: 00036 (2)Single VM with 8 rt vCPUs: Test started at: 2019-10-01 14:31:28 Tuesday Kernel cmdline: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 root=/dev/mapper/rhel_vm--74--64-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm--74--64-swap rd.lvm.lv=rhel_vm-74-64/root rd.lvm.lv=rhel_vm-74-64/swap skew_tick=1 isolcpus=1,2,3,4,5,6,7,8 intel_pstate=disable nosoftlockup nohz=on nohz_full=1,2,3,4,5,6,7,8 rcu_nocbs=1,2,3,4,5,6,7,8 default_hugepagesz=1G iommu=pt intel_iommu=on kvm-intel.vmentry_l1d_flush=never X86 debug pts: pti_enable= ibpb_enabled= ibrs_enabled= retp_enabled= Machine: vm-74-64.lab.eng.pek2.redhat.com CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Test duration(plan): 24h Test ended at: 2019-10-02 14:31:30 Wednesday cyclictest cmdline: taskset -c 1,2,3,4,5,6,7,8 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 8 -a 1,2,3,4,5,6,7,8 --notrace cyclictest results: # Min Latencies: 00008 00009 00009 00014 00007 00009 00009 00009 # Avg Latencies: 00011 00014 00014 00014 00014 00014 00014 00015 # Max Latencies: 00029 00021 00021 00021 00021 00021 00022 00050 (3)Multiple VMs each with 1 rt vCPU: - VM1 Test started at: 2019-10-02 18:35:19 Wednesday Kernel cmdline: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 root=/dev/mapper/rhel_vm--74--34-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm--74--34-swap rd.lvm.lv=rhel_vm-74-34/root rd.lvm.lv=rhel_vm-74-34/swap skew_tick=1 isolcpus=1 intel_pstate=disable nosoftlockup nohz=on nohz_full=1 rcu_nocbs=1 default_hugepagesz=1G iommu=pt intel_iommu=on kvm-intel.vmentry_l1d_flush=never X86 debug pts: pti_enable= ibpb_enabled= ibrs_enabled= retp_enabled= Machine: vm-74-34.lab.eng.pek2.redhat.com CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Test duration(plan): 24h Test ended at: 2019-10-03 18:35:22 Thursday cyclictest cmdline: taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 1 -a 1 --notrace cyclictest results: # Min Latencies: 00007 # Avg Latencies: 00010 # Max Latencies: 00038 - VM2 Test started at: 2019-10-02 18:35:19 Wednesday Kernel cmdline: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 root=/dev/mapper/rhel_bootp--73--75--221-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_bootp--73--75--221-swap rd.lvm.lv=rhel_bootp-73-75-221/root rd.lvm.lv=rhel_bootp-73-75-221/swap skew_tick=1 isolcpus=1 intel_pstate=disable nosoftlockup nohz=on nohz_full=1 rcu_nocbs=1 default_hugepagesz=1G iommu=pt intel_iommu=on kvm-intel.vmentry_l1d_flush=never X86 debug pts: pti_enable= ibpb_enabled= ibrs_enabled= retp_enabled= Machine: bootp-73-75-221.lab.eng.pek2.redhat.com CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Test duration(plan): 24h Test ended at: 2019-10-03 18:35:22 Thursday cyclictest cmdline: taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 1 -a 1 --notrace cyclictest results: # Min Latencies: 00006 # Avg Latencies: 00010 # Max Latencies: 00035 - VM3 Test started at: 2019-10-02 18:35:19 Wednesday Kernel cmdline: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 root=/dev/mapper/rhel_vm--74--204-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm--74--204-swap rd.lvm.lv=rhel_vm-74-204/root rd.lvm.lv=rhel_vm-74-204/swap skew_tick=1 isolcpus=1 intel_pstate=disable nosoftlockup nohz=on nohz_full=1 rcu_nocbs=1 default_hugepagesz=1G iommu=pt intel_iommu=on kvm-intel.vmentry_l1d_flush=never X86 debug pts: pti_enable= ibpb_enabled= ibrs_enabled= retp_enabled= Machine: vm-74-204.lab.eng.pek2.redhat.com CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Test duration(plan): 24h Test ended at: 2019-10-03 18:35:22 Thursday cyclictest cmdline: taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 1 -a 1 --notrace cyclictest results: # Min Latencies: 00007 # Avg Latencies: 00011 # Max Latencies: 00039 - VM4 Test started at: 2019-10-02 18:35:20 Wednesday Kernel cmdline: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 root=/dev/mapper/rhel_vm--74--147-root ro console=tty0 console=ttyS0,115200n8 biosdevname=0 crashkernel=auto resume=/dev/mapper/rhel_vm--74--147-swap rd.lvm.lv=rhel_vm-74-147/root rd.lvm.lv=rhel_vm-74-147/swap skew_tick=1 isolcpus=1 intel_pstate=disable nosoftlockup nohz=on nohz_full=1 rcu_nocbs=1 default_hugepagesz=1G iommu=pt intel_iommu=on kvm-intel.vmentry_l1d_flush=never X86 debug pts: pti_enable= ibpb_enabled= ibrs_enabled= retp_enabled= Machine: vm-74-147.lab.eng.pek2.redhat.com CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Test duration(plan): 24h Test ended at: 2019-10-03 18:35:23 Thursday cyclictest cmdline: taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 1 -a 1 --notrace cyclictest results: # Min Latencies: 00007 # Avg Latencies: 00011 # Max Latencies: 00036 Thank you. Best regards, Pei I've re-test with kernel-rt-4.18.0-147.rt24.93.el8.x86_64 again, this is the 3rd run. The max latency is 39us(the last item 3 below). It may can confirm the solution: "-i 200" has better latency result.
(Continued)
Version Max_Latency Interval_Value
1. kernel-rt-4.18.0-147.rt24.93.el8.x86_64 50us default (copy from description)
2. kernel-rt-4.18.0-147.rt24.93.el8.x86_64 41us -i 200 (copy from description)
3. kernel-rt-4.18.0-147.rt24.93.el8.x86_64 39us -i 200 (re-test again)
Automation beaker job of item 3, please check more details of step if needed:
https://beaker.engineering.redhat.com/jobs/3840824
https://beaker.engineering.redhat.com/jobs/3840826
https://beaker.engineering.redhat.com/jobs/3840906
Pei, I think it's too soon to try to achieve < 20us in RHEL8.2. The kernel is just not ready yet. I think this BZ should be closed for now. (In reply to Luiz Capitulino from comment #4) > Pei, > > I think it's too soon to try to achieve < 20us in RHEL8.2. The kernel is > just not ready yet. > I think this BZ should be closed for now. Hi Luiz, I reported this BZ as it's maybe same issue with rhel7.8: Bug 1761664. If we plan to fix on rhel7.8, maybe we can also apply the fix on rhel8 using same patch. Because "-i 200" can help reduce the max latency. Besides, make sense and it's ok for me to close this bz. Thanks. Best regards, Pei (In reply to Pei Zhang from comment #5) > (In reply to Luiz Capitulino from comment #4) > > Pei, > > > > I think it's too soon to try to achieve < 20us in RHEL8.2. The kernel is > > just not ready yet. > > I think this BZ should be closed for now. > > Hi Luiz, > > I reported this BZ as it's maybe same issue with rhel7.8: Bug 1761664. If we > plan to fix on rhel7.8, maybe we can also apply the fix on rhel8 using same > patch. I see but, RHEL8 doesn't have the patch yet, but we already have a BZ to track it: Bug 1723502 - spurious ktimersoftd wake ups increases latency (rhel-rt 8) I think this one should be closed for now. We've been achieving < 20us in 8.2 (eg. see bug 1757165 comment 28), so closing this one as a dupe. *** This bug has been marked as a duplicate of bug 1757165 *** |
Description of problem: Testing cyclictest with default interval value, the max latency value will be larger than with "-i 200". Version-Release number of selected component (if applicable): kernel-rt-4.18.0-147.rt24.93.el8.x86_64 tuned-2.12.0-3.el8.noarch qemu-kvm-4.1.0-13.module+el8.1.0+4313+ef76ec61.x86_64 libvirt-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64 How reproducible: Sometimes Steps to Reproduce: 1. Setup RT host 2. Setup RT guest 3. Start kernel compiling in host with $twice_of_host_housekeeping_cores 4. Start kernel compiling in guest with $twice_of_guest_housekeeping_cores 5. Start cyclictest in guest with default interval value. # taskset -c 1 /home/nfv-virt-rt-kvm/tools/cyclictest -m -n -q -p95 -D 24h -h60 -t 1 -a 1 -b40 --notrace Step 3,4,5 are executing at same time. Actual results: We collect all of our past rhel8.1 testing results here, the Max_Latency is the max latency value of 24h with 3 standard kvm-rt testing scenarios. Seems "-i 200" has better or at least equal results than default interval value. Version Max_Latency Interval_Value kernel-rt-4.18.0-129.rt24.74.el8.x86_64 54us default kernel-rt-4.18.0-134.rt24.79.el8.x86_64 45us default kernel-rt-4.18.0-137.rt24.83.el8.x86_64 56us default kernel-rt-4.18.0-141.rt24.87.el8.x86_64 48us default kernel-rt-4.18.0-145.rt24.91.el8.x86_64 60us default kernel-rt-4.18.0-146.rt24.92.el8.x86_64 60us default kernel-rt-4.18.0-147.rt24.93.el8.x86_64 50us default kernel-rt-4.18.0-147.rt24.93.el8.x86_64 41us -i 200 Expected results: Testing with default interval value, the cyclictest max latency should not be worse than "-i 200". Additional info: 1. We file a same bz on rhel7.8: Bug 1761664.