Bug 1052900 - [origin_node_374] abused gear will not be throttled
Summary: [origin_node_374] abused gear will not be throttled
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jhon Honce
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-14 11:05 UTC by Meng Bo
Modified: 2015-05-14 23:33 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-30 00:55:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Meng Bo 2014-01-14 11:05:51 UTC
Description of problem:
Create gear on the node, make sure the watchman is running.
Use some shell script to use high cpu on the gear, check the /var/log/messages and cannot find the gear being throttled.

Version-Release number of selected component (if applicable):
fork_ami_origin_node_374_985

How reproducible:
always

Steps to Reproduce:
1. Create a gear on the node
2. Use shell script to burn the cpu
$ dd of=/dev/null if=/dev/zero
3. Check if the gear is throttled

Actual results:
The gear will not be throttled here

Expected results:
The gear should be throttled since abuse

Additional info:
The watchman is running and the throttle_plugin is there,

[root@ip-10-30-170-99 ~]# ps -ef|grep watchman
root      6208     1  0 05:26 ?        00:00:03 watchman                                   
root     28845  3716  0 05:50 pts/0    00:00:00 scl enable ruby193 ruby /usr/sbin/oo-watchman 
root     28850 28847 15 05:50 pts/0    00:00:01 ruby /usr/sbin/oo-watchman
root     28880  3716  0 05:50 pts/0    00:00:00 grep watchman



[root@ip-10-30-170-99 ~]# cat /var/log/messages
Jan 14 05:26:39 ip-10-30-170-99 watchman[6208]: Starting throttler => throttle at: 30.00%, restore at: 70.00%, period: 120, check_interval: 5.00
Jan 14 05:26:39 ip-10-30-170-99 watchman[6208]: Starting Watchman => delay: 20s
Jan 14 05:50:29 ip-10-30-170-99 oo-watchman[28850]: Starting throttler => throttle at: 30.00%, restore at: 70.00%, period: 120, check_interval: 5.00
Jan 14 05:50:29 ip-10-30-170-99 oo-watchman[28850]: Starting Watchman => delay: 20s

[root@ip-10-30-170-99 ~]# ls /etc/openshift/watchman/plugins.d/
jboss_plugin.rb  monitored_gear.rb  stop_lock_plugin.rb  syslog_plugin.rb  throttler_plugin.rb

Comment 1 Jhon Honce 2014-01-14 23:22:02 UTC
Fixed in fork_ami_origin_node_374 988

Comment 2 Meng Bo 2014-01-15 07:11:05 UTC
Checked on fork_ami_origin_node_374_988 issue has been fixed.

Abused gear will be throttled and be restored when silent.

Jan 15 02:05:47 ip-10-236-146-75 watchman[16622]: Throttler: throttle => 52d62f1b41d678047f00001f (96.84)
Jan 15 02:06:07 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.093))
Jan 15 02:06:27 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.168))
Jan 15 02:06:47 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.249))
Jan 15 02:07:07 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.324))
Jan 15 02:07:27 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.41))
Jan 15 02:07:47 ip-10-236-146-75 watchman[16622]: Throttler: restore => 52d62f1b41d678047f00001f (40.045)
Jan 15 02:07:47 ip-10-236-146-75 watchman[16622]: Throttler: throttle => 52d62f1b41d678047f00001f (40.045)
Jan 15 02:08:08 ip-10-236-146-75 watchman[16622]: Throttler: restore => 52d62f1b41d678047f00001f (0.671)

Move bug to verified.


Note You need to log in before you can comment on or make changes to this bug.