Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1068627

Summary: implement lazy save/restore of debug registers
Product: Red Hat Enterprise Linux 7 Reporter: Paolo Bonzini <pbonzini>
Component: kernelAssignee: Paolo Bonzini <pbonzini>
kernel sub component: KVM QA Contact: Virtualization Bugs <virt-bugs>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: alex.williamson, juzhang, knoel, michen, pbonzini, rbalakri, shu, virt-maint, xfu
Version: 7.0   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-3.10.0-143.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 11:40:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1116936    
Bug Blocks:    

Description Paolo Bonzini 2014-02-21 14:21:41 UTC 
Right now, KVM takes a vmexit for each debug register access.  However, debug register accesses usually come in batches of 15-20 accesses, and at ~0.5 microseconds per access they quickly add up.

We can batch debug accesses by setting a flag on the first access and synchronizing all accesses on the next vmexit (whatever the reason for that vmexit is).
Comment 1 Alex Williamson 2014-02-21 15:06:47 UTC 
The game Borderlands 2 running in a VM with an assigned Quadro GPU is a good test of this problem.  Instructions for enabling the console in the game can be found here:

http://forums.gearboxsoftware.com/showpost.php?p=2763900&postcount=1

After enabling, start the game and enable FPS display with 'stat fps' in the game console.  The current FPS and time per frame are shown in the upper right side of the screen.  Start the game and note the FPS.  If we avoid debug register access exits, the FPS will double.  A dirty tracking, lazy restore implementation should see nearly similar results.  The hv-time cpu option is also useful for tuning this application.
Comment 2 Paolo Bonzini 2014-03-25 16:57:03 UTC 
Should also include commits 8246bf52c75aa9b9b336a84f31ed2248754d0f71 and 73aaf249ee2287b4686ff079dcbdbbb658156e64 to bring debug register support in par with upstream.

Regarding testing, there is a testcase in kvm-unit-tests' vmexit test (mov_to_dr) that should see a large improvement after the patches.
Comment 4 Jarod Wilson 2014-08-07 20:54:25 UTC 
Patch(es) available on kernel-3.10.0-143.el7
Comment 7 Shaolong Hu 2014-11-21 05:32:38 UTC 
Trying to test this with GPU passthrough, but hit:

Bug 1163757 - GPU passthrough with Quadro K5000 on HP Z620 host fails to work


I am wondering any other way to test his?

Bests,
Comment 8 Paolo Bonzini 2014-11-24 11:24:26 UTC 
If you run the vmexit.flat test from kvm-unit-tests, the value for the "mov_dr" test will be higher in RHEL7.0 than RHEL7.1 (lower is better).
Comment 9 Shaolong Hu 2014-11-27 03:17:20 UTC 
All vmexit.flat related test, not sure which is key value:

3.10.0-142.el7.x86_64:

qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append cpuid
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
cpuid 3549
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append vmcall
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
vmcall 3439
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append mov_from_cr8
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
mov_from_cr8 11
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append mov_to_cr8
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
mov_to_cr8 15
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append inl_from_pmtimer
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
inl_from_pmtimer 20422
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 2 -append ipi
enabling apic
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
ipi 10661
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 2 -append ipi_halt
enabling apic
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append ple_round_robin
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
Return value from qemu: 1


kernel-3.10.0-205.el7.x86_64:

qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append cpuid
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
cpuid 3142
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append vmcall
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
vmcall 2986
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append mov_from_cr8
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
mov_from_cr8 11
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append mov_to_cr8
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
mov_to_cr8 15
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append inl_from_pmtimer
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
inl_from_pmtimer 19984
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 2 -append ipi
enabling apic
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
ipi 10015
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 2 -append ipi_halt
enabling apic
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
Return value from qemu: 1
qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1 -append ple_round_robin
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
Return value from qemu: 1
Comment 10 Paolo Bonzini 2014-12-01 16:53:11 UTC 
Can you try running it without -append? If you do not get mov_dr, you need an updated vmexit.flat.
Comment 11 Shaolong Hu 2014-12-03 05:13:54 UTC 
3.10.0-205.el7.x86_64:

# qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
cpuid 4551
vmcall 4442
mov_from_cr8 11
mov_to_cr8 15
inl_from_pmtimer 39678
inl_from_qemu 39892
inl_from_kernel 15182
outl_to_kernel 6164
mov_dr 118
...


3.10.0-142.el7.x86_64:

# qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/vmexit.flat -smp 1
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
pci-testdev at 0x20 membar febf1000 iobar c000
cpuid 3351
vmcall 3251
mov_from_cr8 11
mov_to_cr8 15
inl_from_pmtimer 20103
inl_from_qemu 20082
inl_from_kernel 6421
outl_to_kernel 3846
mov_dr 3357
...


118 VS 3357, lower is better, verified.
Comment 13 errata-xmlrpc 2015-03-05 11:40:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0290.html