Bug 1184155
Summary: | Dynamic tickless feature not working in RHEL7 KVM guest | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Sunil Choudhary <schoudha> |
Component: | kernel | Assignee: | Marcelo Tosatti <mtosatti> |
kernel sub component: | KVM | QA Contact: | Virtualization Bugs <virt-bugs> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | high | ||
Priority: | high | CC: | drjones, hhuang, huding, juzhang, knoel, lcapitulino, mazhang, mtosatti, riel, scui, tbowling, virt-maint, xfu |
Version: | 7.2 | ||
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | kernel-3.10.0-262.el7 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-11-19 21:15:47 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1193629 | ||
Bug Blocks: | 1133060 |
Description
Sunil Choudhary
2015-01-20 17:05:17 UTC
Patch posted upstream: http://marc.info/?l=kvm&m=142982357227117&w=2 Patch(es) available on kernel-3.10.0-262.el7 Steps to Reproduce: # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-123.el7.x86_64 root=/dev/mapper/rhel_hp--z220--01-root ro rd.lvm.lv=rhel_hp-z220-01/root biosdevname=0 console=tty0 reboot=pci console=ttyS0,115200 crashkernel=auto rd.lvm.lv=rhel_hp-z220-01/swap rhgb quiet LANG=en_US.UTF-8 systemd.debug # perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [11093] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [11093] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 1,000 irq_vectors:local_timer_entry 1.000820275 seconds time elapsed Verify on 3.10.0-262.el7.x86_64 # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-262.el7.x86_64 root=/dev/mapper/rhel_hp--z220--01-root ro rd.lvm.lv=rhel_hp-z220-01/root biosdevname=0 console=tty0 reboot=pci console=ttyS0,115200 crashkernel=auto rd.lvm.lv=rhel_hp-z220-01/swap rhgb quiet LANG=en_US.UTF-8 systemd.debug nohz_full=1-7 isolcpus=1-7 # perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [11153] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [11153] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 5 irq_vectors:local_timer_entry 1.000817904 seconds time elapsed But cannot see 1 tick, please correct me if I'm wrong. Host cpu infos: processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 58 model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz stepping : 9 microcode : 0x1b cpu MHz : 1817.804 cache size : 8192 KB physical id : 0 siblings : 8 core id : 3 cpu cores : 4 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt bogomips : 6784.67 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: Thanks, Shuping I'll let Marcelo give the final answer on this, but although I can't tell why local_timer_entry() is being called 5 times per second, I think that the requested feature _is_ working as we're down from 1,000 periodic calls to a few sporadic calls. Maybe we could open a new BZ to debug those calls, and move this one forward. But that's small issue, IMHO. Also, I have a few more comments: 1. Keep in mind that even when full tickless is taking place, the kernel will setup a minimum of 1 timer interrupt per second. This is so because the scheduler must maintain some stats 2. When testing Marcelo fixes a few months ago, I'd see that KVM's timer function in the host (which emulates the tick) would be called twice a second (instead of once, as explained above). I haven't debduged this yet, but I don't think it's a huge issue either 3. Maybe local_timer_entry() is being called due to timers being setup in that CPU? (In reply to Luiz Capitulino from comment #11) > I'll let Marcelo give the final answer on this, but although I can't tell > why local_timer_entry() is being called 5 times per second, I think that the > requested feature _is_ working as we're down from 1,000 periodic calls to a > few sporadic calls. Maybe we could open a new BZ to debug those calls, and > move this one forward. But that's small issue, IMHO. > > Also, I have a few more comments: > > 1. Keep in mind that even when full tickless is taking place, the kernel > will setup a minimum of 1 timer interrupt per second. This is so because the > scheduler must maintain some stats > > 2. When testing Marcelo fixes a few months ago, I'd see that KVM's timer > function in the host (which emulates the tick) would be called twice a > second (instead of once, as explained above). I haven't debduged this yet, > but I don't think it's a huge issue either > > 3. Maybe local_timer_entry() is being called due to timers being setup in > that CPU? stress uses "alarm" and "nanosleep" syscalls, both of them make use of timers. Test this bug on 3.10.0-261.el7.x86_64 and 3.10.0-300.el7.x86_64 Result: 3.10.0-261.el7.x86_64 [root@dhcp-11-16 ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-261.el7.x86_64 root=/dev/mapper/rhel_dhcp--11--16-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-11-16/root rd.lvm.lv=rhel_dhcp-11-16/swap nohz_full=1-7 isolcpus=1-7 LANG=en_US.UTF-8 [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [10808] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [10808] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 6 irq_vectors:local_timer_entry 1.023972406 seconds time elapsed [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [14390] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [14390] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 12 irq_vectors:local_timer_entry 1.000963412 seconds time elapsed [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [14426] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [14426] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 8 irq_vectors:local_timer_entry 1.000885183 seconds time elapsed 3.10.0-300.el7.x86_64 [root@dhcp-11-16 ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-300.el7.x86_64 root=/dev/mapper/rhel_dhcp--11--16-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-11-16/root rd.lvm.lv=rhel_dhcp-11-16/swap nohz_full=1-7 isolcpus=1-7 LANG=en_US.UTF-8 [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [14689] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [14689] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 10 irq_vectors:local_timer_entry 1.001913108 seconds time elapsed [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [14692] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [14692] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 7 irq_vectors:local_timer_entry 1.000826672 seconds time elapsed [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [14695] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [14695] successful run completed in 1s Performance counter stats for 'CPU(s) 1': 9 irq_vectors:local_timer_entry 1.000824091 seconds time elapsed Seems there are no difference between 3.10.0-261.el7.x86_64 and 3.10.0-300.el7.x86_64. Marcelo, please correct me if the test steps is wrong. Thanks, Mazhang. Host info: processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 58 model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz stepping : 9 microcode : 0x1b cpu MHz : 2442.421 cache size : 8192 KB physical id : 0 siblings : 8 core id : 3 cpu cores : 4 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt bogomips : 6784.48 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: [root@dhcp-11-16 ~]# free -m total used free shared buff/cache available Mem: 7766 300 6903 8 562 7234 Swap: 7935 0 7935 (In reply to mazhang from comment #15) > Test this bug on 3.10.0-261.el7.x86_64 and 3.10.0-300.el7.x86_64 > > Result: > 3.10.0-261.el7.x86_64 > > [root@dhcp-11-16 ~]# cat /proc/cmdline > BOOT_IMAGE=/vmlinuz-3.10.0-261.el7.x86_64 > root=/dev/mapper/rhel_dhcp--11--16-root ro crashkernel=auto > rd.lvm.lv=rhel_dhcp-11-16/root rd.lvm.lv=rhel_dhcp-11-16/swap nohz_full=1-7 > isolcpus=1-7 LANG=en_US.UTF-8 > [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset > -c 1 stress -t 1 -c 1 > stress: info: [10808] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd > stress: info: [10808] successful run completed in 1s > > Performance counter stats for 'CPU(s) 1': > > 6 irq_vectors:local_timer_entry > > 1.023972406 seconds time elapsed > > [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset > -c 1 stress -t 1 -c 1 > stress: info: [14390] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd > stress: info: [14390] successful run completed in 1s > > Performance counter stats for 'CPU(s) 1': > > 12 irq_vectors:local_timer_entry > > 1.000963412 seconds time elapsed > > [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset > -c 1 stress -t 1 -c 1 > stress: info: [14426] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd > stress: info: [14426] successful run completed in 1s > > Performance counter stats for 'CPU(s) 1': > > 8 irq_vectors:local_timer_entry > > 1.000885183 seconds time elapsed > > > 3.10.0-300.el7.x86_64 > > [root@dhcp-11-16 ~]# cat /proc/cmdline > BOOT_IMAGE=/vmlinuz-3.10.0-300.el7.x86_64 > root=/dev/mapper/rhel_dhcp--11--16-root ro crashkernel=auto > rd.lvm.lv=rhel_dhcp-11-16/root rd.lvm.lv=rhel_dhcp-11-16/swap nohz_full=1-7 > isolcpus=1-7 LANG=en_US.UTF-8 > [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset > -c 1 stress -t 1 -c 1 > stress: info: [14689] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd > stress: info: [14689] successful run completed in 1s > > Performance counter stats for 'CPU(s) 1': > > 10 irq_vectors:local_timer_entry > > > 1.001913108 seconds time elapsed > > [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset > -c 1 stress -t 1 -c 1 > stress: info: [14692] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd > stress: info: [14692] successful run completed in 1s > > Performance counter stats for 'CPU(s) 1': > > 7 irq_vectors:local_timer_entry > > > 1.000826672 seconds time elapsed > > [root@dhcp-11-16 ~]# perf stat -C 1 -e irq_vectors:local_timer_entry taskset > -c 1 stress -t 1 -c 1 > stress: info: [14695] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd > stress: info: [14695] successful run completed in 1s > > Performance counter stats for 'CPU(s) 1': > > 9 irq_vectors:local_timer_entry > > > 1.000824091 seconds time elapsed > > > Seems there are no difference between 3.10.0-261.el7.x86_64 and > 3.10.0-300.el7.x86_64. > > Marcelo, please correct me if the test steps is wrong. > > Thanks, > Mazhang. Yes it is correct. You might want to check the others vCPUs as well. Reproduce this bug on kernel-3.10.0-123.el7.x86_64. Host: 3.10.0-300.el7.x86_64 qemu-kvm-rhev-2.1.2-16.el7.x86_64 Guest: 3.10.0-123.el7.x86_64 Result: # uname -r 3.10.0-123.el7.x86_64 # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-123.el7.x86_64 root=UUID=ded051b2-66ad-40f8-b601-509a061dd98a ro rd.lvm.lv=rhel_virtlab-66-84-114/root console=tty0 vconsole.keymap=us rd.lvm.lv=rhel_virtlab-66-84-114/swap console=ttyS0,115200 crashkernel=auto vconsole.font=latarcyrheb-sun16 biosdevname=0 nohz_full=1-3 isolcpus=1-3 # perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [3119] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [3119] successful run completed in 1s Performance counter stats for 'taskset -c 1 stress -t 1 -c 1': 999 irq_vectors:local_timer_entry 1.001063742 seconds time elapsed Update guest kernel to 3.10.0-300.el7.x86_64 and re-test. Result: # uname -r 3.10.0-300.el7.x86_64 # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-300.el7.x86_64 root=/dev/mapper/rhel_virtlab--66--84--114-root ro rd.lvm.lv=rhel_virtlab-66-84-114/root console=tty0 vconsole.keymap=us rd.lvm.lv=rhel_virtlab-66-84-114/swap console=ttyS0,115200 crashkernel=auto vconsole.font=latarcyrheb-sun16 biosdevname=0 nohz_full=1-3 isolcpus=1-3 LANG=en_US.UTF-8 # perf stat -C 1 -e irq_vectors:local_timer_entry taskset -c 1 stress -t 1 -c 1 stress: info: [3010] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd stress: info: [3010] successful run completed in 1s Performance counter stats for 'taskset -c 1 stress -t 1 -c 1': 7 irq_vectors:local_timer_entry 1.025816153 seconds time elapsed This bug has been fixed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-2152.html |