Bug 608981

Also observed extremely poor scheduler latency relative to prior kernel.  Going back to old version.

tested kernel 2.6.18-194.8.1.el5

bug is still present

tested kernel 2.6.18-194.11.1.el5

problem still there

However just loaded 2.6.18-194.11.1.el5 on a HP
DL165 and the problem is not present.  Difference
between the Tyan and the HP is that Tyan has
Nvidia chipset and HP has Broadcom/Serverworks
chipset.  So probably it has something to do with
the Nvidia chipset support drivers.

Exact chipset is nVIDIA NFP3600.

http://tyan.com/product_board_spec.aspx?pid=157

Created attachment 441744 [details]
lspci output

Created attachment 441745 [details]
lspci -vvv output

I found turning off C2+C3 states in the BIOS has eliminated the CPU double-accounting problem on a couple different AMD servers we have.  Don't know much about the side-effects of doing this, but in three cases, performance improved significantly with the ondemand governor as some cores never increased their clock rate when C2+C3 was enabled.

ACPI power management has always been disabled on our Tyan S2912 where the problem appears.

Hello Red Hat!  Any plans to ever look into this rather serious regression?

(In reply to comment #7)
> ACPI power management has always been disabled on our Tyan S2912 where the
> problem appears.
> 
> Hello Red Hat!  Any plans to ever look into this rather serious regression?

Regression versus what? Have you identified a kernel version where this worked?

The 194.x.y kernels aren't likely to have a fix for your issue just suddenly crop up, as these updates are typically for cve's and panic fixes. The development kernels are located elsewhere. If you're looking to try something newer that might have a fix, wander over here:

http://people.redhat.com/~jwilson/el5/

Hmm.  Seemed so obvious that it never occurred to me
to state it.  Every kernel prior to RHEL 5.5 works fine,
the most recent we tried, and still run in production is

    2.6.18-164.6.1.el5

Don't want non-prod release.  We'll stick with old
kernel till it works.

(In reply to comment #9)
> Hmm.  Seemed so obvious that it never occurred to me
> to state it.  Every kernel prior to RHEL 5.5 works fine,
> the most recent we tried, and still run in production is
> 
>     2.6.18-164.6.1.el5

Okay, that's very helpful to know, thank you.

> Don't want non-prod release.  We'll stick with old
> kernel till it works.

Hrm, okay. Well, to have a shot at fixing this, I think we'll have to run some non-prod releases *somewhere*, be that on your end or on ours, if we can dig up a machine that reproduces the problem. Based on comment #3, it would seem to be nvidia-chipset-specific.

Prarit, is there someone in your team who would be an ideal candidate to dig into what seems to be an nvidia chipset issue? Not sure this should still be assigned to Bhavna, since it might not be an AMD issue, per se.

> Prarit, is there someone in your team who would be an ideal candidate to dig
> into what seems to be an nvidia chipset issue? Not sure this should still be
> assigned to Bhavna, since it might not be an AMD issue, per se.

It seems like everyone on the team's had their fingers in NVIDIA ;)  Jaroslav, Michal, me, etc..

I'll cc a few people and ask them to take a look.

P.

Any chance you could do a 'dmidecode' and send us the output?  I'd like to see if we have a similar system available in our test lab.

Thanks,

P.

Created attachment 452391 [details]
'dmidecode' output from Tyan B2912 server

>Hrm, okay. Well, to have a shot at fixing this, I think we'll have to run some
>non-prod releases *somewhere*, be that on your end or on ours, if we can dig up
>a machine that reproduces the problem. Based on comment #3, it would seem to be
>nvidia-chipset-specific.

I am willing to try out fixes that are intended to specifically correct
this issue--it's easy enough to fire up a kernel, run 'top' and 
the simple shell loop during off-hours.  Just don't want to be stabbing
in the dark running random kernels hoping they will work.

Also willing to run a debug/trace setup to help identify the
issue if it's not a huge pain.

(In reply to comment #16)
> >Hrm, okay. Well, to have a shot at fixing this, I think we'll have to run some
> >non-prod releases *somewhere*, be that on your end or on ours, if we can dig up
> >a machine that reproduces the problem. Based on comment #3, it would seem to be
> >nvidia-chipset-specific.
> 
> I am willing to try out fixes that are intended to specifically correct
> this issue--it's easy enough to fire up a kernel, run 'top' and 
> the simple shell loop during off-hours.  Just don't want to be stabbing
> in the dark running random kernels hoping they will work.

We're not planning on asking you to do that.

However, we may ask you boot and run some kernels in order for us to narrow down the specific build that this problem occurred in.

What was the last pre-5.5 kernel you booted that you know worked?

P.

as stated, 2.6.18-164.6.1.el5

(In reply to comment #16)
> >Hrm, okay. Well, to have a shot at fixing this, I think we'll have to run some
> >non-prod releases *somewhere*, be that on your end or on ours, if we can dig up
> >a machine that reproduces the problem. Based on comment #3, it would seem to be
> >nvidia-chipset-specific.
> 
> I am willing to try out fixes that are intended to specifically correct
> this issue--it's easy enough to fire up a kernel, run 'top' and 
> the simple shell loop during off-hours.  Just don't want to be stabbing
> in the dark running random kernels hoping they will work.

Yeah, no, not what I was meaning to suggest. But giving the latest 5.6 development kernel a quick spin could be enlightening. Its *possible* the issue has already been fixed, as we're several hundred patches newer than the 5.5 kernel by now, which includes some timer-related fixes and various process accounting patches. None of them, from what I recall, specifically said they'd fix your very issue, but its possible they did. If so, we may be able to simply cherry-pick the fix for a 5.5 build. If not, we know we've also got to fix 5.6 -- we don't want to fix it for 5.5, only to have the problem crop back up in 5.6.

Ok, I can try the new kernel sometime in next week or so to see if it fixes it.

Are there no Opteron servers at RH with the Nvidia chipset?  Both Supermicro and Tyan made a bunch of them--thousands or tens of thousands I'm sure.  Probably HP did not, but I wouldn't be surprised if Dell or IBM did.

(In reply to comment #20)
> Ok, I can try the new kernel sometime in next week or so to see if it fixes it.
> 
> Are there no Opteron servers at RH with the Nvidia chipset?  

We do have them.  The concern is that it may be a combination of the mobo + chipset + processor that is causing the problem.  That specific set of hardware will likely be difficult to track down.

Thanks for you offer of helping with debugging -- we appreciate that we're asking you to test on production servers and that isn't an easy thing to do.

P.

Tried 2.6.18-225.el5 and the problem is corrected in this kernel.

Sorry for being skeptical here.  The upstream guys will
waste days of one's time with no thought or consideration,
and I've been burned by this a couple of times when
attempting to try out vanilla kernels--truly a bad idea.
The result is that I'm a bit cautious.

Created attachment 452472 [details]
full 'dmesg' output

Curious 'dmesg' entries that might be relevent:

NFORCE-MCP55: IDE controller at PCI slot 0000:00:04.0
NFORCE-MCP55: chipset revision 161
NFORCE-MCP55: not 100% native mode: will probe irqs later
NFORCE-MCP55: BIOS didn't set cable bits correctly. Enabling workaround.
NFORCE-MCP55: 0000:00:04.0 (rev a1) UDMA133 controller

'dmesg' from the buggy kernel version for would be interesting to compare.
Perhaps it's using a different clocksource? Do the contents of /sys/devices/system/clocksource/*/current_clocksource differ between the kernels?

will post it tomorrow

how long does it take to reproduce the problem? Does it show up immediately in top or does it take a while? So far the only thing I see is:
Tasks: 227 total,   2 running, 225 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.3%us,  0.0%sy,  0.0%ni, 91.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4051336k total,  1537740k used,  2513596k free,   106936k buffers
Swap:  6094840k total,        0k used,  6094840k free,  1177296k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4584 root      25   0 66068 1628 1272 R  100  0.0  44:06.51 bash


Could be that the machine (MCP55) I'm using doesn't show the problem, but I want to make sure I'm not missing anything.

Bug is binary, either present or not present.  200% or 100%.  Very simple to reproduce when a system is affected.

Created attachment 452774 [details]
dmesg_tyan_s2912_2.6.18-194.11.1.el5.txt

dmesg output from system affected by 'top' CPU doubled reporting issue

I've tried all kinds of MCP55+Opteron machines we have available, none of them shows the behaviour described.

Could you please test the kernel from
http://people.redhat.com/jwilson/el5/215.el5/

I hope you're okay to test some kernels. It cuts down the amount of commits we need to consider by a lot. Thanks!

Ok.  Will be a couple of days till I can do it.

Did you check with the other individual who has this problem?  Maybe 
they can identify some other systems with the issue.  He did not 
appear to subscribe to the bug so you would have to e-mail him 
directly.

Ok, I get the hint.  Bisected it for you:

kernel-2.6.18-225.el5  good
kernel-2.6.18-215.el5  bad
kernel-2.6.18-222.el5  bad
kernel-2.6.18-223.el5  good

So whatever the patch is that fixes it was added in 223.

Thank you for testing these kernel. Additionally could you please confirm that the kernel from http://people.redhat.com/jwilson/el5/227.el5/ also works?
Testing with this kernel might isolate the responsible patch.

kernel 2.6.18-227.el5 good

Tried running 2.6.18-229.el5 in production and it works fine.  So I suppose generating a patch for 2.6.18-194 can be skipped unless someone else has the problem and is allergic to unsupported kernels.