Bug 841330 - Regression in performance of kernel 3.4.4 vs. 3.3.7 when running as Xen Dom0
Regression in performance of kernel 3.4.4 vs. 3.3.7 when running as Xen Dom0
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
16
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Kernel Maintainer List
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-18 13:28 EDT by W. Michael Petullo
Modified: 2013-02-13 20:29 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-13 20:29:25 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/proc/cpuinfo while running kernel 3.4.4 (3.05 KB, text/plain)
2012-07-18 13:29 EDT, W. Michael Petullo
no flags Details
Output from "xl dmesg" with cpufreq=verbose,performance, but NOT xen-acpi-processor.off=1 (5.43 KB, text/plain)
2012-07-27 18:22 EDT, W. Michael Petullo
no flags Details
Output from "xl dmesg" with cpufreq=verbose,performance, but NOT xen-acpi-processor.off=1; kernel and Xen from latest comment (5.50 KB, text/plain)
2012-11-07 13:24 EST, W. Michael Petullo
no flags Details

  None (edit)
Description W. Michael Petullo 2012-07-18 13:28:14 EDT
Description of problem:
I run Xen with Fedora 16 serving as Dom0. I have found that upgrading to kernel 3.4.4 degrades the performance of both Dom0 (i.e., itself) and DomU (i.e., other kernels) vs. 3.3.7.

Version-Release number of selected component (if applicable):
kernel-3.4.4-4.fc16.x86_64

How reproducible:
Every time

Steps to Reproduce:
1. Boot 3.4.4 as Dom0.
2. Build glibc-2.16.0 with "time make".
3. Boot 3.3.7 and repeat.
  
Actual results:
3.4.4:
    real: 21:25
    user:  5:12
    sys:  16:03

3.3.7:
    real:  9:58
    user:  4:50
    sys:   4:42

Expected results:
3.4.4 and 3.3.7 should have similar performance.

Additional info:
Looking at /proc/cpuinfo, I noticed that 3.3.7 provides flag aperfmperf and 3.4.4 provides hw_pstate (but not vice versa).
Comment 1 W. Michael Petullo 2012-07-18 13:29:38 EDT
Created attachment 598953 [details]
/proc/cpuinfo while running kernel 3.4.4
Comment 2 Josh Boyer 2012-07-18 13:47:54 EDT
Adding Konrad on CC.
Comment 3 W. Michael Petullo 2012-07-20 13:57:30 EDT
I booted Xen using the Linux 3.4.4 kernel while passing "cpufreq=verbose,performance" to Xen. I confirmed that Xen booted using this argument by checking the output of "xl dmesg". I saw slow results similar to those on 3.4.4 above:

real: 21:28
user: 5:16
sys:  16:01
Comment 4 Konrad Rzeszutek Wilk 2012-07-27 13:28:13 EDT
One more thing, just to eliminate this being the power management fault. Can you on the Linux command line boot with 'xen-acpi-processor.off=1'?

Did you this performance regression start with 3.4.0? Did you try to see if removing some of the commits that I posted on xen-devel had any effect?
Comment 5 W. Michael Petullo 2012-07-27 14:04:53 EDT
That does it. Setting 'xen-acpi-processor.off=1' restores the performance I see on 3.3.7.

I have not yet been able to back any of the commits from my kernel.
Comment 6 Konrad Rzeszutek Wilk 2012-07-27 17:10:50 EDT
(In reply to comment #5)
> That does it. Setting 'xen-acpi-processor.off=1' restores the performance I
> see on 3.3.7.

Yeey! So the cpufreq=performance on the Xen hypervisor line _should_ have done it too. Hmm.

When you ran with 'cpufreq=performance,verbose' did you get any extra Xen output? I am quite curious to see what kind of powersave information it had uploaded that either caused the hypervisor to slow down or .. it its a bug in the hypervisor.
Check with 'xl dmesg' please.

I am leaning towards the latter, so what kind of machine is this? What's the motherboard/RAM? Is the hypervisor a custom compiled one or did you just use the default Fedora one?
> 
> I have not yet been able to back any of the commits from my kernel.

That is Ok. The xen-acpi-processor.off=1 is the failsafe I added in case something would go wrong (which it seems it has).

Also, could I ask you to email Xen-devel on the email thread with these findings (and the motheroad/CPU) so that we can CC the AMD folks to see if they know what is up with the hypervisor.

Thanks!
Comment 7 W. Michael Petullo 2012-07-27 18:22:04 EDT
Created attachment 600871 [details]
Output from "xl dmesg" with cpufreq=verbose,performance, but NOT xen-acpi-processor.off=1
Comment 8 W. Michael Petullo 2012-07-27 18:32:17 EDT
Motherboard: ASUS M5A97, BIOS version 0901
Processor: AMD FX 4170 quad core, 4.2GHz, 12.0MB total cache
Memory: 16GB DDR3 1,151MHz

The hypervisior is standard Fedora xen-4.1.2-8.fc16.x86_64.
Comment 9 Fedora Update System 2012-08-08 16:02:06 EDT
xen-4.1.2-10.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/xen-4.1.2-10.fc16
Comment 10 Michael Young 2012-08-08 16:07:43 EDT
I have built xen-4.1.2-10.fc16 which includes the patch
http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053
which it has been suggested might fix this problem.
Comment 11 Fedora Update System 2012-08-11 06:33:37 EDT
xen-4.1.3-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/xen-4.1.3-1.fc16
Comment 12 Fedora Update System 2012-08-12 22:27:04 EDT
Package xen-4.1.3-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing xen-4.1.3-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-11785/xen-4.1.3-1.fc16
then log in and leave karma (feedback).
Comment 13 W. Michael Petullo 2012-08-13 12:41:18 EDT
I tried:

    xen-4.1.3-1.fc16.x86_64
    kernel-3.4.7-1.fc16.x86_64

without any special boot parameters. The glibc build completed as follows:

real	21m5.056s
user	5m16.343s
sys	15m36.680s

So this combination still has a performance regression (vs. 3.3.7 in the initial comment).
Comment 14 W. Michael Petullo 2012-08-13 15:46:09 EDT
Actually, I was too quick on my last comment. I followed the instructions from the Fedora Update System too closely and forgot to update xen-hypervisor, etc (i.e., I only updated the xen package).

Here are my new results:

real	10m44.211s
user	5m2.713s
sys	5m18.508s

This is still a bit slower but much less so, and it may be due to the imprecision of my "benchmark".
Comment 15 Fedora Update System 2012-08-21 05:53:39 EDT
xen-4.1.3-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 16 W. Michael Petullo 2012-11-07 13:22:27 EST
Something again is wrong.

I am using the same machine with:

    kernel-3.6.5-2.fc16.x86_64
    xen-4.1.3-2.fc16.x86_64

When I boot in Xen, I notice a performance degredation. When I boot with "xen-acpi-processor.off=1" all is well.
Comment 17 W. Michael Petullo 2012-11-07 13:24:01 EST
Created attachment 640247 [details]
Output from "xl dmesg" with cpufreq=verbose,performance, but NOT xen-acpi-processor.off=1; kernel and Xen from latest comment
Comment 18 Fedora End Of Life 2013-01-16 17:59:41 EST
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 19 Fedora End Of Life 2013-02-13 20:29:29 EST
Fedora 16 changed to end-of-life (EOL) status on 2013-02-12. Fedora 16 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.