The following has be reported by IBM LTC:  
Up to 6 Second Latencies in some cpu bound transactions
Hardware Environment: Egenera Blades

Software Environment: RH 2.4.9-e25

Steps to Reproduce: Can only be done at CSFB's production environment.
1.  Run CSFB's AMM application (stock quoting system)
2.  Observe 6 second maximum logged transaction time (happens every 15 minutes)

Additional Information:

CSFB's stock quoting application has some cpu bound transactions that take up to
6 seconds, while average time is 24ms.  Kernel 2.4.7-10 did not experience this,
but kernel 2.4.9-e25 does. 

We have instrumented try_to_free_pages to see if we were stuck in a long quest
for memory allocation.  None of the threads called try_to_free_pages during the
test (6 second transactions were still there).  Kswapd also did not call
try_to_free_pages

We have instrumented task timeslice assignments (min, max) per task to look for
scheduler starvation.  All timeslices were 15ms.  All tasks had the same priority.

We are preparing for a A/B test with a "whitebox" system to compare to Egenera's
blade systems.


We would like RedHat's help with this problem:

1) Have there been any situations like this, a multi-second latency for cpu
bound operations?

2) What else can we instrument to better identify the problem

Glen/Greg - this is a performance bug against a Red Hat errata kernel.
Per Andrew's request, please submit this to Red Hat. Thanks.