Bug 1098243
Summary: | Add auto-throttling support for storage client subsystem | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [JBoss] JBoss Operations Network | Reporter: | John Sanda <jsanda> | ||||||||||
Component: | Core Server, Storage Node | Assignee: | Simeon Pinder <spinder> | ||||||||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Armine Hovsepyan <ahovsepy> | ||||||||||
Severity: | medium | Docs Contact: | |||||||||||
Priority: | unspecified | ||||||||||||
Version: | JON 3.2 | CC: | ahovsepy, fbrychta, genman, hrupp, loleary, lzoubek, mfoley, miburman | ||||||||||
Target Milestone: | CR01 | ||||||||||||
Target Release: | JON 3.3.1 | ||||||||||||
Hardware: | Unspecified | ||||||||||||
OS: | Unspecified | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | 1045589 | Environment: | |||||||||||
Last Closed: | 2015-02-27 19:58:22 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Bug Depends On: | 1126209, 1045589, 1106508, 1108095, 1149342 | ||||||||||||
Bug Blocks: | |||||||||||||
Attachments: |
|
Description
John Sanda
2014-05-15 14:42:55 UTC
The upstream bug 1045589 has already been closed, but this needs to be revisited. We have since upgraded the DataStax driver. There may have been some changes in the driver around when it sends event notifications that could impact the throttling. I am setting the target milestone to ER4 to allow for time to revisit this. The changes for this have actually been in the release branch. I left it ASSIGNED though due to ongoing work for other, related bugs. Earlier today while testing I realized that there is an issue that needs to be resolved. When a timeout occurs, StorageSession.java decreases the request limit. The request limit however, will not be increased to its original rate without a server restart. This is bad because timeouts are likely to occur during spikes or bursts in traffic. For example, if a server has been offline for a while, agents will spool measurement reports. When the server comes back online, it will get hammered with those measurement reports. We are more susceptible to timeouts during bursts like this. When that burst or spike in traffic has subsided, we should ideally return to the original request per second rate. Now if there is a pattern of repeated timeouts of a period of time that includes peaks and valleys in request traffic, that is likely an indicator that we either need to, A) decrease the rate limit B) decrease load C) deploy additional nodes I think that dealing with repeated timeouts over a sustained interval is beyond the scope of this bug, but we do need to deal with the temporary spikes so that a server restart is not needed to get back to the original request limit. We will have to tackle this in ER05. Added to master: commit 57c72fd8254998281c157852d0f3bf3e363c09a6 Author: Michael Burman <miburman> Date: Tue Oct 7 12:36:50 2014 +0300 [BZ 1098243] [BZ 1149342] When encountering a timeout on StorageSession, recreate RateLimiter with higher warmup time. branch: release/jon3.3.x commit b0031db6702b26520b92fb30b83774dcb53f86fb Author: Michael Burman <miburman> Date: Tue Oct 7 12:36:50 2014 +0300 [BZ 1098243] [BZ 1149342] When encountering a timeout on StorageSession, recreate RateLimiter with higher warmup time. (cherry picked from commit 57c72fd8254998281c157852d0f3bf3e363c09a6) Signed-off-by: Libor Zoubek <lzoubek> Moving to ON_QA as available to test with the latest brew build: https://brewweb.devel.redhat.com//buildinfo?buildID=394734 missleading description on gui - screenshot attached Created attachment 952512 [details]
throttling-config
I agree that the description should be updated. The property descriptions in rhq-server.properties should also be updated. I agree that the description should be updated. The property descriptions in rhq-server.properties should also be updated. reassigning based on comment #10 passing tcms testcase execution run for throttling https://tcms.engineering.redhat.com/run/191519/ Please clarify whether this is functionally complete? From the looks of it, the only remaining task for this BZ is to fix the property description in the JBoss ON UI and rhq-server.properties file? Target is being set to 3.3.1 as description change would be low risk. Yes, this is functionally complete, but needs updates to the UI & properties-file to remove redundant stuff. Fixed in master, removed the old information and added new configuration options (as well as metric for the current warmup period if user wants to create an alert based on throttling). commit e8a902ed399920177c2cb2bf4801be7ca61309ba Author: Michael Burman <miburman> Date: Wed Jan 7 14:53:30 2015 +0200 [BZ 1098243] Remove redundant request limit configuration options and make the warmup periods configurable through the UI branch: release/jon3.3.x link: https://github.com/rhq-project/rhq/commit/a8cbf18cf time: 2015-01-12 10:42:33 +0100 commit: a8cbf18cf25eb8dfa20886ebc5b7dcc40ce5d06b author: Michael Burman - miburman message: [BZ 1098243] Remove redundant request limit configuration options and make the warmup periods configurable through the UI (cherry picked from commit e8a902ed399920177c2cb2bf4801be7ca61309ba) Signed-off-by: Libor Zoubek <lzoubek> Moving to ON_QA as available for test with the latest 3.3.1.ER01 bits from here: http://download.devel.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/12/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip 1. Fix is visible on GUI -- see screenshot attached 2. rhq-server.properties fix (from https://github.com/rhq-project/rhq/commit/a8cbf18cf) - is not visible after applying patch -- seems not included into patch script -- screenhost attached. reassigning back Created attachment 985159 [details]
rhq-server-properties-331
Created attachment 985160 [details]
throttling-config-gui-331
Moving to ON_QA as available to test with latest CP build: http://download.devel.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/16/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip *Note: jon-server-patch-3.3.0.GA.zip maps to CR01 build of jon-server-3.3.0.GA-update-01.zip. verified in JON 3.3.1 CR1 1. rhq-server.properties.new is created with correct changed 2. applied patch with rhq-server.properties merged with properties.new Created attachment 992109 [details]
rhq-server-properties-331_new
|