Description of problem: Create gear on the node, make sure the watchman is running. Use some shell script to use high cpu on the gear, check the /var/log/messages and cannot find the gear being throttled. Version-Release number of selected component (if applicable): fork_ami_origin_node_374_985 How reproducible: always Steps to Reproduce: 1. Create a gear on the node 2. Use shell script to burn the cpu $ dd of=/dev/null if=/dev/zero 3. Check if the gear is throttled Actual results: The gear will not be throttled here Expected results: The gear should be throttled since abuse Additional info: The watchman is running and the throttle_plugin is there, [root@ip-10-30-170-99 ~]# ps -ef|grep watchman root 6208 1 0 05:26 ? 00:00:03 watchman root 28845 3716 0 05:50 pts/0 00:00:00 scl enable ruby193 ruby /usr/sbin/oo-watchman root 28850 28847 15 05:50 pts/0 00:00:01 ruby /usr/sbin/oo-watchman root 28880 3716 0 05:50 pts/0 00:00:00 grep watchman [root@ip-10-30-170-99 ~]# cat /var/log/messages Jan 14 05:26:39 ip-10-30-170-99 watchman[6208]: Starting throttler => throttle at: 30.00%, restore at: 70.00%, period: 120, check_interval: 5.00 Jan 14 05:26:39 ip-10-30-170-99 watchman[6208]: Starting Watchman => delay: 20s Jan 14 05:50:29 ip-10-30-170-99 oo-watchman[28850]: Starting throttler => throttle at: 30.00%, restore at: 70.00%, period: 120, check_interval: 5.00 Jan 14 05:50:29 ip-10-30-170-99 oo-watchman[28850]: Starting Watchman => delay: 20s [root@ip-10-30-170-99 ~]# ls /etc/openshift/watchman/plugins.d/ jboss_plugin.rb monitored_gear.rb stop_lock_plugin.rb syslog_plugin.rb throttler_plugin.rb
Fixed in fork_ami_origin_node_374 988
Checked on fork_ami_origin_node_374_988 issue has been fixed. Abused gear will be throttled and be restored when silent. Jan 15 02:05:47 ip-10-236-146-75 watchman[16622]: Throttler: throttle => 52d62f1b41d678047f00001f (96.84) Jan 15 02:06:07 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.093)) Jan 15 02:06:27 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.168)) Jan 15 02:06:47 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.249)) Jan 15 02:07:07 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.324)) Jan 15 02:07:27 ip-10-236-146-75 watchman[16622]: Throttler: REFUSED restore => 52d62f1b41d678047f00001f (still over threshold (263.41)) Jan 15 02:07:47 ip-10-236-146-75 watchman[16622]: Throttler: restore => 52d62f1b41d678047f00001f (40.045) Jan 15 02:07:47 ip-10-236-146-75 watchman[16622]: Throttler: throttle => 52d62f1b41d678047f00001f (40.045) Jan 15 02:08:08 ip-10-236-146-75 watchman[16622]: Throttler: restore => 52d62f1b41d678047f00001f (0.671) Move bug to verified.