Bug 264481

Summary: Recent kernel update killed overall performance
Product: [Fedora] Fedora Reporter: Matt Fago <fago>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: medium    
Version: 6   
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2007-09-11 16:57:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matt Fago 2007-08-29 17:51:09 UTC
Description of problem:

I was out of the office for the past two months or so. When I returned I did a "yum update" in fedora. 
This pulled down a new version of the kernel. VMWare (after patching) is now unbelievably slow (it 
takes several minutes for the application to open, much less start a virtual machine). My own software 
(straight C code terminal app, no GUI etc) is also about 25% slower. Was there a new scheduler or 
something?

Version-Release number of selected component (if applicable):

Unknown -- from an update in the past 9 weeks.

How reproducible:

100%

Steps to Reproduce:
1. Run my own executable under "time", or launch VMWare (version 5.x)

  
Actual results:

My own executable takes roughly 25% longer to run, while VMWare (the app itself, not a virtual 
machine) is almost unusable.

Expected results:

The same performance as 9 weeks ago.

Additional info:

This might be difficult to pin-down. Could someone help me downgrade the kernel so I could identify 
the update that broke everything?

Comment 1 Chuck Ebbert 2007-08-31 18:18:14 UTC
(In reply to comment #0)
> Steps to Reproduce:
> 1. Run my own executable under "time", 

What is the output, under the new and old kernels?

And can you try this kernel?

http://people.redhat.com/cebbert/kernels/FC6/kernel-2.6.22.5-49.fc6.x86_64.rpm

Boot into the 2.6.20 kernel and remove the previous 2.6.22 one, then install
this one (using rpm). That way there will only be one 2.6.22 version installed.




Comment 2 Matt Fago 2007-08-31 18:27:13 UTC
I don't have the exact values, but it is roughly:

old kernel, user: 3m 25s
new kernel, user: 4m 45s

I will not have access to the machine until next week. How should I remove the previous 2.6.22 kernel?




Comment 3 Chuck Ebbert 2007-08-31 19:10:09 UTC
We need the complete output of the 'time' command. (And the user time reported
may just be an artifact of the way the new sceduler does accounting, does it
really take more wall clock time?)

To erase the old kernel (as root)

  # rpm -e kernel-2.6.22.whatever.fc6

Install the new one (all on one line):

  # rpm -i
http://people.redhat.com/cebbert/kernels/FC6/kernel-2.6.22.5-49.fc6.x86_64.rpm



Comment 4 Matt Fago 2007-08-31 19:40:27 UTC
Ah, of course. Thanks. Yum has spoiled me.

Here is the output of the 'time' command with the current kernel (I do not have one saved from 9 weeks 
ago):

real    4m52.590s
user    4m41.283s
sys     0m0.411s

The old executions looked very similar, but took roughly user 3m25s, real slightly larger.

My application does seem to take more wall-clock time, and VMWare certainly is much much slower.

I'll try the new kernel as soon as I have local access to the machine.



Comment 5 Matt Fago 2007-09-04 21:45:32 UTC
Here is the timing with version 2.6.22.5-49

real    4m34.799s
user    4m30.528s
sys     0m0.350s

I haven't tried to recompile the VMWare drivers to see if it helps with that.

 - Matt



Comment 6 Matt Fago 2007-09-04 21:56:13 UTC
Oh, by the way I only had 2.6.22.1 and 2.6.22.2 installed. What was the last
version of the FC6 x86_64 kernel with the old scheduler so I can try that again?

Thanks,
Matt

Comment 7 Chuck Ebbert 2007-09-04 22:10:53 UTC
(In reply to comment #6)
> Oh, by the way I only had 2.6.22.1 and 2.6.22.2 installed. What was the last
> version of the FC6 x86_64 kernel with the old scheduler so I can try that again?
> 

kernel-2.6.20-1.2962.fc6



Comment 8 Chuck Ebbert 2007-09-05 20:58:22 UTC
kernel 2.6.22.5-49, in the updates-testing repo, has the latest scheduler updates.

Comment 9 Matt Fago 2007-09-07 01:31:41 UTC
Ah, what info do you need?

I'm planning to revert to kernel-2.6.20-1.2962.fc6 and test as soon as I have local access to the machine 
again (probably next Tuesday).

Other than this, I'd assume it's fallout from Ingo's CFS merge? Seems like it might be "fair" to the 
detriment of a heavy computational task. Something tunable? Worth firing a message to LKML? If so, I'd 
rather someone more versed in kernel details do so if possible.

Comment 10 Chuck Ebbert 2007-09-07 13:15:41 UTC
(In reply to comment #9)
> Ah, what info do you need?
> 

Results from 2.6.20 and from 2.6.22.5-49, which has the latest CFS scheduler.
2.6.22.5-49 is in updates-testing:

# yum --enablerepo=updates-testing update kernel

To keep more kernels installed than the default of 2, edit
/etc/yum/pluginconf.d/installonlyn.conf and change 'tokeep=2' to some larger number.


Comment 11 Matt Fago 2007-09-07 14:42:45 UTC
Resuts from 2.6.22.5-49 are in comment #5. Very small improvement. I'll get 2.6.20 next week.

Comment 12 Matt Fago 2007-09-11 16:52:18 UTC
Reinstalling 2.6.20-1.2962 resulted in no change:

real    4m38.239s
user    4m28.565s
sys     0m0.266s

Thus it seems like it is something other than the kernel. I suppose this bug
should be closed in some fashion. I'll open a new bug if I can figure out what
caused the slowdown I'm seeing.

Thanks for all of your help,
Matt