Description of problem:

There are two significant errors in the math in the MonitoredGear class used by the throttler plugin.

1) The code calculates the total possible cpu time by multiplying nr_periods by cfs_quota_us.  However, nr_periods is only a counter of the periods when the cgroup actually ran something (or wanted to?).  It is _not_ a count of elapsed time.

2) The number calculated with (nr_periods*cfs_quota_us) is in microseconds, but the throttled time and cpuacct.usage values against which it is compared are in nanoseconds.  The percentage calculation would thus be off by a factor of 1000, except the conversion from a raw number to a percent was commented out, so it off by a less-noticeable (???) factor of 10.

Version-Release number of selected component (if applicable):

openshift-origin-node-util-1.23.4-1.el6oso.noarch

How reproducible:

Always

Steps to Reproduce:

Problem 1:

1. On a mostly-idle node, run a process in a gear that is compute-intensive.  Here's one:

ruby -e 'z=0; 1.upto(1_000_000_000_000) { |x| 1.upto(x) { |y| z += y } }'

2. Watch the gear get throttled, and note that the logs say things like:

May 22 17:24:53 ex-std-node1 watchman[345260]: Throttler: REFUSED restore => 537b749b2587c85c1c0001ec (still over threshold (999.98))

Problem 2:

1. On a mostly-idle node, run a process in a gear that is compute-intensive, but in bursts.  E.g.:

while true; do time ruby -e 'z=0; 1.upto(1000) { |x| 1.upto(x) { |y| z += y } }'; sleep 5; done

I specifically timed this so the command is using about 1 second of CPU time in my environment, and thus the throttler should measure it at about 16% utilization.  Because of its use of nr_periods, it typically says 80-90% instead.

Actual results:

Problem 1: The gear appears to be using more than 100% of its CPU allocation, which 1) suggests that cgroups don't actually work, and 2) prevents gears from being unthrottled when they should be.

Problem 2: Gears with bursty traffic but low overall CPU utilization are often penalized and throttled.

Expected results:

The throttler's calculations should more accurately reflect the actual percentage of CPU consumed.

Additional info:

I have a small patch on the way for this, and a script for testing as well.