RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 645898 - [6.1 FEAT] Port KVM bug fixes for cr_access to RHEL 6
Summary: [6.1 FEAT] Port KVM bug fixes for cr_access to RHEL 6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.1
Hardware: other
OS: All
medium
medium
Target Milestone: rc
: 6.1
Assignee: Karen Noel
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 538808 580566 Rhel6KvmTier1 658891
TreeView+ depends on / blocked
 
Reported: 2010-10-22 20:00 UTC by IBM Bug Proxy
Modified: 2013-01-09 23:16 UTC (History)
17 users (show)

Fixed In Version: kernel-2.6.32-83.el6
Doc Type: Enhancement
Doc Text:
Prior to this update, running context-switch intensive workloads on KVM guests resulted in a large number of exits (kvm_exit) due to control register (CR) accesses by the guest, thus, resulting in poor performance. This update includes a number of optimizations which allow the guest not to exit to the hypervisor in the aforementioned case and improve the overall performance.
Clone Of:
Environment:
Last Closed: 2011-05-23 20:27:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
patch 1 of 3 (1.17 KB, text/plain)
2010-10-29 22:21 UTC, IBM Bug Proxy 		no flags Details
patch 2 of 3 (2.54 KB, text/plain)
2010-10-29 22:21 UTC, IBM Bug Proxy 		no flags Details
patch 3 of 3 (16.74 KB, text/plain)
2010-10-29 22:21 UTC, IBM Bug Proxy 		no flags Details
patch 1 of 3 (16.74 KB, text/plain)
2010-11-01 16:21 UTC, IBM Bug Proxy 		no flags Details
patch 2 of 3 (2.54 KB, text/plain)
2010-11-01 16:21 UTC, IBM Bug Proxy 		no flags Details
patch 3 of 3 (1.17 KB, text/plain)
2010-11-01 16:21 UTC, IBM Bug Proxy 		no flags Details
test program causing lots of cr_access exits (768 bytes, text/x-csrc)
2011-02-22 10:03 UTC, Avi Kivity 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 67906 0 None None None Never
Red Hat Product Errata RHSA-2011:0542 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 6.1 kernel security, bug fix and enhancement update 2011-05-19 11:58:07 UTC

Description IBM Bug Proxy 2010-10-22 20:00:35 UTC 
1. Feature Overview:
Feature Id: [67906]
a. Name of Feature: [6.1 FEAT] Port KVM bug fixes for cr_access to RHEL 6
b. Feature Description
Guest performance of the DayTrader benchmark suffers because of too many guest exits due to CR0
access.  A KVM trace during the peak load of the benchmark had 36% of its kvm_exit entries due to
cr_access.

Avi Kivity has made several commits to the KVM upstream kernel which virtually eliminate the
kvm_exit entries due to cr_access.  See commits:
bfb3f3326c915b1800dc65d10ca09fbd548353d2
50c3a8197508798b9b004a7832935faed4a90678
8bdf7ac877b865fa7affa47ab6d38fbabe5881d5
b6aa9639f71221a97e430056ba26adb68f9a377f
6a1b7a29a9ec09ae5f81f763896bee7778dbccb4
2f701cb3ffaba103a6c3e0a090c89d8765640cf1
34d8621e84352b376b253f35de8395975a7e72dd

These fixes should be ported to RHEL 6.1

Running with RHEL 6.0 rc1 kernel patched with these fixes gave a 10% increase in the DayTrader score
over the RHEL 6.0 rc1 kernel.  The KVM trace taken during the peak load had no kvm_exit entries for
cr_access.

The same problem (many exits due to CR0 access) has been seen on the SPECjEnterprise benchmark.


2. Feature Details:
Sponsor: LTC Performance
Architectures:  x86, x86_64, 

Arch Specificity: purely arch specific code
Affects Kernel Modules: Yes
Delivery Mechanism: Direct from Community
Category: kernel
Request Type: Kernel - Enhancement from Upstream
d. Upstream Acceptance: Accepted
Sponsor Priority P3
f. Severity: normal
IBM Confidential: No
Code Contribution: no
g. Component Version Target: ---

3. Business Case
These fixes improve the performance of KVM, making it a more competitive solution for our customers.

4. Primary contact at Red Hat:
John Jarvis, jjarvis

5. Primary contacts at Partner:
Project Management Contact:
Michael L. Hohnbaum, hbaum.com

Technical contact(s):
ANTHONY N. LIGUORI, aliguori.com
Comment 2 IBM Bug Proxy 2010-10-29 22:21:09 UTC 
Created attachment 456570 [details]
patch 1 of 3


------- Comment (attachment only) From kmr.com 2010-10-29 18:16 EDT-------
Comment 3 IBM Bug Proxy 2010-10-29 22:21:13 UTC 
Created attachment 456571 [details]
patch 2 of 3


------- Comment (attachment only) From kmr.com 2010-10-29 18:16 EDT-------
Comment 4 IBM Bug Proxy 2010-10-29 22:21:17 UTC 
Created attachment 456572 [details]
patch 3 of 3


------- Comment (attachment only) From kmr.com 2010-10-29 18:16 EDT-------
Comment 5 IBM Bug Proxy 2010-11-01 16:21:07 UTC 
Created attachment 456909 [details]
patch 1 of 3


------- Comment on attachment From kmr.com 2010-11-01 12:12 EDT-------


Correct patch sequencing and filename.
Comment 6 IBM Bug Proxy 2010-11-01 16:21:16 UTC 
Created attachment 456910 [details]
patch 2 of 3


------- Comment on attachment From kmr.com 2010-11-01 12:12 EDT-------


Correct patch sequencing and filename.
Comment 7 IBM Bug Proxy 2010-11-01 16:21:30 UTC 
Created attachment 456911 [details]
patch 3 of 3


------- Comment on attachment From kmr.com 2010-11-01 12:13 EDT-------


Correct patch sequencing and filename.
Comment 8 Aristeu Rozanski 2010-11-17 19:46:52 UTC 
Patch(es) available on kernel-2.6.32-83.el6
Comment 19 IBM Bug Proxy 2010-11-30 21:11:25 UTC 
------- Comment From mcintire.com 2010-11-30 16:10 EDT-------
*** Bug 67653 has been marked as a duplicate of this bug. ***
Comment 23 John Jarvis 2010-12-01 14:52:31 UTC 
This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux minor release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux Update minor release.   
While it is a goal to include this enhancement in the next minor release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next minor release pending the next phase of actual 
code integration and successful Red Hat and partner testing.
Comment 29 Miya Chen 2011-02-17 10:05:18 UTC 
Removed "OtherQA" keywords as QE can test this bug inside after confirming with dev.
Comment 30 Mike Cao 2011-02-21 07:55:26 UTC 
Compared the times kvm_exit entries due to cr_access between
2.6.32-71.el6.x86_64 and 2.6.32-117.el6.x86_64.

Test on following steps:
1.on the host 
#mount -t debugfs none /sys/kernel/debug
#echo 1 >/sys/kernel/debug/tracing/events/kvm/enable
2.start a linux guest (also tried start a windows guest with cpu load via
vmprime.exe)
3.check how many times kvm_exit entries due to cr_access in 5 mins
#cat /sys/kernel/debug/tracing/trace_pipe|grep kvm_exit|grep cr_access|wc -l
following is the results :

Actual Resuts:
/--------------------------------------\
|                          |   rhel6   |
|--------------------------|-----------|
|2.6.32-71.el6.x86_64      |  440      |
|--------------------------|-----------|
|2.6.32-117.el6.x86_64     |  176      |
\--------------------------------------/

the times kvm_exit entries due to cr_access is smaller in 2.6.32-117.el6.x86_64 than that in 2.6.32-71.el6.x86_64 

Based on above, this issue has been fixed.
Comment 31 IBM Bug Proxy 2011-02-21 19:51:07 UTC 
------- Comment From steved.com 2011-02-21 14:48 EDT-------
Those are awfully low cr_access counts for five minutes of activity.  Was anything run in the guest, or did it just sit idle?  IIRC, the problem surfaces when there was a lot of task switching.  An idle guest won't expose the problem and thus won't prove if it is fixed.
Comment 32 Avi Kivity 2011-02-22 10:03:56 UTC 
Created attachment 480089 [details]
test program causing lots of cr_access exits

Please try the attached test program.

$ gcc -O2 -pthread -o cr_access-test cr_access-test.c
$ taskset 1 ./cr_access-test

I see double the throughput with kernel that has the fixes.

The program is for Linux, but should compile on Windows with mingw.
Comment 33 IBM Bug Proxy 2011-02-23 02:51:25 UTC 
------- Comment From habanero.com 2011-02-22 21:48 EDT-------
I have tested the errata kernel (2.6.32-71.17)  with server consolidation benchmark and have observed the expect 4% increase in performance.
Comment 34 Martin Prpič 2011-02-23 15:04:48 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Prior to this update, running context-switch intensive workloads on KVM guests resulted in a large number of exits (kvm_exit) due to control register (CR) accesses by the guest, thus, resulting in poor performance. This update includes a number of optimizations which allow the guest not to exit to the hypervisor in the aforementioned case and improve the overall performance.
Comment 35 IBM Bug Proxy 2011-04-14 21:11:27 UTC 
------- Comment From aliguori.com 2011-04-14 17:02 EDT-------
Using Avi's test program, I'm able to verify that this change increases the number of CR accesses between RHEL 6.0 (2.6.32-30.el6.x86_64) and RHEL6.1 (2.6.32-125.el6.x86_64) by a factor of 4x.
Comment 36 IBM Bug Proxy 2011-04-14 21:52:13 UTC 
------- Comment From steved.com 2011-04-14 17:40 EDT-------
An "increase [in the] number of CR accesses between RHEL 6.0 (2.6.32-30.el6.x86_64) and RHEL6.1
(2.6.32-125.el6.x86_64) by a factor of 4x" sounds like a bad thing.  The idea was to decrease the number of CR accesses.  Does this mean RHEL 6.1 regressed?

------- Comment From aliguori.com 2011-04-14 17:47 EDT-------
The test case measures the number of cr accesses per second within the guest.  More cs accesses (in the guest) per second implies a cheaper cost of doing a cr access.  IOW, more is good.
Comment 37 Mike Cao 2011-04-15 02:26:51 UTC 
Thanks For aliguori's work!

Based on comment #35, comment #36 ,
Change status to VERIFIED.
Comment 38 errata-xmlrpc 2011-05-23 20:27:10 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html
Comment 39 IBM Bug Proxy 2012-03-26 19:01:11 UTC 
------- Comment From steved.com 2012-03-26 14:54 EDT-------
*** Bug 67652 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.