Bug 995550 - Application initial build in devenv being throttled - means boosting is cancelled
Application initial build in devenv being throttled - means boosting is cance...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Fotios Lindiakos
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-09 13:09 EDT by Clayton Coleman
Modified: 2015-05-14 19:25 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-29 08:50:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Clayton Coleman 2013-08-09 13:09:43 EDT
In the devenv, the application initial build is being throttled which is causing creation to time out.  We made changes a few sprints ago to give initial builds more CPU, and it looks like the throttler needs to skip throttling when boosting is in play.
Comment 1 Fotios Lindiakos 2013-08-09 15:47:46 EDT
Modified the way we look for gears during throttle. This will also ensure that we do not throttle a boosted app.

PR submitted, undergoing manual testing now

https://github.com/openshift/origin-server/pull/3328
Comment 2 openshift-github-bot 2013-08-09 20:02:30 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/3ccbe91d0aa799cf4c9817509d3026833773f215
Merge pull request #3328 from fotioslindiakos/Bug995550

Merged by openshift-bot
Comment 3 Meng Bo 2013-08-16 02:54:19 EDT
Tested the throttler on devenv-stage_442,

The app after creation has default cgroup setting for now.

But the auto-throttler does not work now.

Steps:
1.Create a php app
2.Make the app using high rate CPU
3.Watch the /var/log/messages
4.Watch the gear cgroup settings

Following logs found in /var/log/messages
Aug 16 02:45:31 ip-10-203-70-75 rhc-watchman[1950]: Throttler: restore => 520dc585b9ba120aee000001 (0.117)
Aug 16 02:45:31 ip-10-203-70-75 rhc-watchman[1950]: Throttler: REFUSED restore => 165896602154797635731456 (still over threshold)
Aug 16 02:45:31 ip-10-203-70-75 rhc-watchman[1950]: Throttler: REFUSED throttle => 165896602154797635731456 (gear already throttled)
Aug 16 02:45:51 ip-10-203-70-75 rhc-watchman[1950]: Running rhc-watchman => delay: 20s, exception threshold: 10
Aug 16 02:45:51 ip-10-203-70-75 rhc-watchman[1950]: Throttler: restore => 520dc585b9ba120aee000001 (0.137)
Aug 16 02:45:51 ip-10-203-70-75 rhc-watchman[1950]: Throttler: REFUSED restore => 165896602154797635731456 (still over threshold)
Aug 16 02:45:51 ip-10-203-70-75 rhc-watchman[1950]: Throttler: REFUSED throttle => 165896602154797635731456 (gear already throttled)
Aug 16 02:46:11 ip-10-203-70-75 rhc-watchman[1950]: Running rhc-watchman => delay: 20s, exception threshold: 10
Aug 16 02:46:11 ip-10-203-70-75 rhc-watchman[1950]: Throttler: restore => 520dc585b9ba120aee000001 (0.091)
Aug 16 02:46:11 ip-10-203-70-75 rhc-watchman[1950]: Throttler: REFUSED restore => 165896602154797635731456 (still over threshold)
Aug 16 02:46:11 ip-10-203-70-75 rhc-watchman[1950]: Throttler: REFUSED throttle => 165896602154797635731456 (gear already throttled)


But the gear is actually in default cgroup setting.
#oo-cgroup-template -c 165896602154797635731456
165896602154797635731456: default
# cat /cgroup/all/openshift/165896602154797635731456/cpu.cfs_quota_us 
100000


That means the app will not be auto trottled, and all the throttle events are refused.

BTW, the auto restore works well after manual set the cgroup to throttle for gear.
Aug 16 02:53:32 ip-10-203-70-75 rhc-watchman[1950]: Running rhc-watchman => delay: 20s, exception threshold: 10
Aug 16 02:53:32 ip-10-203-70-75 rhc-watchman[1950]: Throttler: restore => 520dc585b9ba120aee000001 (0.139)
Aug 16 02:53:32 ip-10-203-70-75 rhc-watchman[1950]: Throttler: restore => 165896602154797635731456 (0.099)
Comment 4 Fotios Lindiakos 2013-08-20 17:59:37 EDT
Fix in this PR, undergoing review and testing.

https://github.com/openshift/origin-server/pull/3443
Comment 5 Meng Bo 2013-08-22 01:52:59 EDT
Checked on devenv-stage_448.

Watch the app cgroup setting when gear is creating. The gear will be temporary boosted and will not be throttled automatically.

And found log about the boosted gear will not be throttled after gear created.


Aug 22 01:44:20 ip-10-196-51-239 rhc-watchman[1928]: Throttler: REFUSED throttle => 342483957415281973788672 (gear is boosted)
Aug 22 01:44:40 ip-10-196-51-239 rhc-watchman[1928]: Running rhc-watchman => delay: 20s, exception threshold: 10
Aug 22 01:44:40 ip-10-196-51-239 rhc-watchman[1928]: Throttler: REFUSED throttle => 342483957415281973788672 (gear is boosted)
Aug 22 01:45:00 ip-10-196-51-239 rhc-watchman[1928]: Running rhc-watchman => delay: 20s, exception threshold: 10
Aug 22 01:45:00 ip-10-196-51-239 rhc-watchman[1928]: Throttler: REFUSED throttle => 342483957415281973788672 (gear is boosted)
Aug 22 01:45:20 ip-10-196-51-239 rhc-watchman[1928]: Running rhc-watchman => delay: 20s, exception threshold: 10


Gear can be throttled after set the gear back to default cgroup template.

Aug 22 01:47:40 ip-10-196-51-239 rhc-watchman[1928]: Running rhc-watchman => delay: 20s, exception threshold: 10
Aug 22 01:47:40 ip-10-196-51-239 rhc-watchman[1928]: Throttler: throttle => 342483957415281973788672 (521.298)
Aug 22 01:48:00 ip-10-196-51-239 rhc-watchman[1928]: Running rhc-watchman => delay: 20s, exception threshold: 10
Aug 22 01:48:00 ip-10-196-51-239 rhc-watchman[1928]: Throttler: REFUSED restore => 342483957415281973788672 (still over threshold (628.67))

Move the bug to Verified.

Note You need to log in before you can comment on or make changes to this bug.