Description of problem: Unable to display metrics WARN [org.rhq.coregui.server.gwt.MeasurementDataGWTServiceImpl] (http-/0.0.0.0:8443-9) Sending exception to client: [1401794902338] : javax.ejb.EJBException: java.lang.IllegalArgumentException: lowValue (270.0) is not less than or equal to value (135.13888888888889). at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:191) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:237) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:299) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:212) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.rhq.enterprise.server.measurement.MeasurementDataManagerLocal$$$view123.findDataForResource(Unknown Source) [rhq-server.jar:4.9.0.JON320GA-redhat-1] at org.rhq.coregui.server.gwt.MeasurementDataGWTServiceImpl.findDataForResource(MeasurementDataGWTServiceImpl.java:104) [classes:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_21] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_21] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_21] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_21] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561) [gwt-servlet-2.5.0.jar:] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208) [gwt-servlet-2.5.0.jar:] at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248) [gwt-servlet-2.5.0.jar:] at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) [gwt-servlet-2.5.0.jar:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1] at org.rhq.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:88) [classes:] ... Caused by: java.lang.IllegalArgumentException: lowValue (270.0) is not less than or equal to value (135.13888888888889). at org.rhq.core.domain.measurement.composite.MeasurementDataNumericHighLowComposite.<init>(MeasurementDataNumericHighLowComposite.java:49) [rhq-core-domain-ejb3.jar:4.9.0.JON320GA-redhat-1] at org.rhq.server.metrics.MetricsServer.createComposites(MetricsServer.java:375) [rhq-server-metrics-4.9.0.JON320GA.jar:4.9.0.JON320GA] at org.rhq.server.metrics.MetricsServer.findDataForResource(MetricsServer.java:182) [rhq-server-metrics-4.9.0.JON320GA.jar:4.9.0.JON320GA] at org.rhq.enterprise.server.measurement.MeasurementDataManagerBean.findDataForResource(MeasurementDataManagerBean.java:844) [rhq-server.jar:4.9.0.JON320GA-redhat-1] ... Version-Release number of selected component (if applicable): 3.2.1 How reproducible: Depending on the data and date range, this can happen fairly often and consistently.
Here is a brief summary of how this bug can occur. It is triggered with late measurement reports. By late I mean that aggregation has already been run for the reported times. For example, suppose the current time is 15:10, and the server receives a report with data having timestamps from the 13:00 hour. We already aggregated data from the 13:00; therefore, the measurement report is late. Now let's say that the server is restarted about a week later. At start up we search for the most recent raw data to determine if there was any data prior to the server shutdown that needs to be aggregated. Due to a small bug in that start up check, we determine that data from 13:00 a week ago needs to be aggregated. The retention period for raw data is only 7 days; so, when we go to query for it, we will get back an empty result set. The method that is called to aggregate data for a particular schedule id does not expect an empty result set. It produces an aggregate metric where min and max are NaN and avg is zero. This aggregate metric replaces the previous one from the 13:00 hour and consequently leads to a 6 hr aggregate where min > avg. It should be noted that it could also produce an aggregate metric where max < avg.
Here is a more detailed explanation. At server start up StorageClientManagerBean calls MetricsServer.init(). That method in turn calls determineMostRecentRawDataSinceLastShutdown(), and it performs the check for the most recent raw data. This is done by querying the metrics_index table. Rows in metrics_index are keyed by time slice, e.g., 2014-07-01 08:00:00, 2014-07-01 09:00:00, etc. When we insert data, we execute two writes - one to the data table and one to metrics_index. There is a bug in determineMostRecentRawDataSinceLastShutdown() where we do not check the current hour. We instead start with the previous hour and query metrics_index as far back as previous hour - 7 days. If MetricsServer finds past data that needs to be aggregated, it sets a flag so that the data will be aggregated during the next aggregation run. When the data purge job runs, MetricsServer.calculateAggregates() is invoked. It queries the metrics_index table to find out which schedule ids have data to be aggregated. When the data aggregation for a time slice is finished, the corresponding row in metrics_index is deleted. That row can and will get recreated with late data. It is assumed that there is data for the schedule ids returned from the metrics_index query. That assumption is of course part of the problem. MetricsServer calls calculateaggregateRawData(DateTime theHour). This method queries metrics_index, iterates through the result set, queries the raw_data table for each schedule id, and calls calculateAggregatedRaw(Iterable<RawNumericMetric> rawMetrics, long timestamp). This method looks like, // start calculateAggregatedRaw double min = Double.NaN; double max = min; int count = 0; ArithmeticMeanCalculator mean = new ArithmeticMeanCalculator(); double value; for (RawNumericMetric metric : rawMetrics) { ... } return new AggregateNumericMetric(0, mean.getArithmeticMean(), min, max, timestamp); // end calculateAggregatedRaw Because rawMetrics is empty, the min and max are Double.NaN and the avg is zero. This can lead to invalid 6 hr metrics. Suppose we had the following 1 hr aggregate metrics, {time: 06:00, max: 100, min: 100, avg: 100} {time: 07:00, max: 100, min: 100, avg: 100} {time: 08:00, max: 100, min: 100, avg: 100} {time: 09:00, max: 100, min: 100, avg: 100} {time: 10:00, max: 100, min: 100, avg: 100} {time: 11:00, max: 100, min: 100, avg: 100} Now let's say we had a late measurement report for the 10:00 and the data is not aggregated until a week later. This results in, {time: 10:00, max: NaN, min: NaN, avg: 0} and will result in this 6 hr aggregate metric, {time: 06:00, max: 100, min: 100, avg: 83.33} This is an invalid metric since min > avg. We can prevent this from happening by only calling calculateAggregatedRaw() and calculateAggregate() when data is returned for the schedule id during aggregation.
For existing metrics, I suggest we check for invalid data on query methods. When we we find an invalid metric, * Exclude it from the results returned to the client * Log a warning * Try to recalculate the metric (Note that this may not be possible if the data from which it was generated has already expired) * If the metric cannot be recomputed, delete it
Here are steps to reproduce that can be used for QE verification. To keep things simple, use a default installation with server and storage node running on the same box. 1. Use the REST API to insert a value of 100 for a particular schedule id during the 12:00 hour 2. Repeat step 1 during the 13:00, 14:00, 15:00, 16:00, and 17:00 hours. 3. After the data purge job finishes in the 18:00 hour, re-insert data for the 14:00 hour. 4. Shut down the server and storage node. 5. Set the system clock ahead to one week later at 16:00. 6. Restart the server and storage node. 7. When the data purge job runs, we will recalculate aggregates for the 14:00 hour from a week ago and should produce and this invalid 6 hr metric, {time: 12:00, max: 100, min: 100, avg: 83.33} 8. Query from either REST API or UI to generate exception reported in the initial description. The start date for the query can be now, and the end date needs to be greater than 2 weeks and less than 31 days.
Work for the steps described in comment 4 has been done at https://github.com/jsanda/rhq/tree/bug/1110462. I have merged a squash commit of that work to the release/jon3.2.x branch. commit hash: 6445f3a6412529 When an invalid metric is detected, the server will log a message like this, 22:11:53,147 WARN [org.rhq.server.metrics.MetricsServer] (http-/0.0.0.0:7080-133) The one_hour_metrics metric AggregatedNumericMetric[scheduleId=10006, avg=1.0384E10, min=1.0385268736E10, max=1.0385268736E10, timestamp=1404867600000] is invalid. It will be excluded from the results sent to the client and we will attempt to recompute the metric. If the metric can be recomputed, then the server will log a message like this, 22:24:10,368 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-23-thread-1) The invalid 1 hour metric AggregatedNumericMetric[scheduleId=10006, avg=1.0384E10, min=1.0385268736E10, max=1.0385268736E10, timestamp=1404867600000] has been recomputed with a new value of {max: 1.0385268736E10, min: 1.0385268736E10, avg: 1.0385268736E10} If the server cannot recompute the metric, InvalidMetricsManager will log a message indicating that it was deleted. Invalid metrics are not processed immediately. An invalid metric is added to a queue, and after a delay of 10 minutes, it is processed. There is a background thread that checks the queue every 5 minutes. Every time the thread runs, it drains the queue of each metric whose delayed has expired. The delay is there to prevent redundant work that could result as a result of graphs refreshing for example.
Moving to ON_QA as available for test with build: http://jon01.mw.lab.eng.bos.redhat.com:8042/dist/release/jon/3.2.2.GA/7-11-14_0600/
Tried to push the data to a schedule using REST API, however I failed to push the data. When I checked the URL for raw metric data for the schedule, ( Ex: http://10.65.201.105:7080/rest/metric/data/11504/raw.html ), I am getting 'JBWEB000065: HTTP Status 500' error in the UI and in server log. Discussed with jsanda about the error, seems like the update needs to include the rest WAR. Please refer the attached screen-shot for the error in UI and the stacktrace in server log.
Created attachment 918149 [details] screenshot
Created attachment 918150 [details] stacktrace
Update 2 needs to includes a recompiled rhq-rest.war. Sunil hit a NoSuchMethodError because code that is called in the REST API has changed. The signature of a method has changed which apparently is causing the NoSuchMethodError.
Moving to ON_QA as available for test with 3.2 Update 02 CR02 build: http://jon01.mw.lab.eng.bos.redhat.com:8042/dist/release/jon/3.2.2.GA/7-16-2014/
Verified on JBoss ON 3.2.2 Update 02 CR02 build. Followed the steps mentioned. When queried using UI for start date as now and end date greater than 2 weeks and less than 31 days, verified that the invalid metric is recomputed with a new avg value. The server log has below: 16:15:33,685 WARN [org.rhq.server.metrics.MetricsServer] (http-/0.0.0.0:7080-1) The six_hour_metrics metric AggregatedNumericMetric[scheduleId=11566, avg=83.33, min=100.0, max=100.0, timestamp=1405665000000] is invalid. It will be excluded from the results sent to the client and we will attempt to recompute the metric. 16:15:33,701 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (http-/0.0.0.0:7080-1) Adding InvalidMetric{type=6 hour, scheduleId=11566, timestamp=1405665000000, max=100.0, min=100.0, avg=83.33} to queue for processing 16:29:55,633 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) Deleting 6 hour metric AggregatedNumericMetric[scheduleId=11566, avg=83.33, min=100.0, max=100.0, timestamp=1405665000000] since the 1 hour metrics are no longer available. 16:29:55,648 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) InvalidMetric{type=6 hour, scheduleId=11566, timestamp=1405665000000, max=100.0, min=100.0, avg=83.33} has been recomputed with a new value of {max: 100.0, min: 100.0, avg: 100.0} Verified that the avg value in UI for the metric also shows the recomputed value.
This has been verified and released in Red Hat JBoss Operations Network 3.2 Update 02 (3.2.2) available from the Red Hat Customer Portal[1]. [1]: https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=31783
Reopening as this issue is still occurring in 3.2.2. ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:8443-88) JBAS014134: EJB Invocation failed on component MeasurementDataManagerBean for method public abstract java.util.List org.rhq.enterprise.server.measurement.MeasurementDataManagerRemote.findDataForResource(org.rhq.core.domain.auth.Subject,int,int[],long,long,int): javax.ejb.EJBException: java.lang.IllegalArgumentException: lowValue (3.771219968E9) is not less than or equal to value (3.2094454095238094E9). at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:191) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:237) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:299) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:212) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] ... at org.rhq.enterprise.server.measurement.MeasurementDataManagerLocal$$$view123.findDataForResource(Unknown Source) [rhq-server.jar:4.9.0.JON320GA-redhat-2] at org.rhq.coregui.server.gwt.MeasurementDataGWTServiceImpl.findDataForResource(MeasurementDataGWTServiceImpl.java:104) ... at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1] at org.rhq.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:88) ... Caused by: java.lang.IllegalArgumentException: lowValue (3.771219968E9) is not less than or equal to value (3.2094454095238094E9). at org.rhq.core.domain.measurement.composite.MeasurementDataNumericHighLowComposite.<init>(MeasurementDataNumericHighLowComposite.java:49) [rhq-core-domain-ejb3.jar:4.9.0.JON320GA-redhat-2] at org.rhq.server.metrics.MetricsServer.createComposites(MetricsServer.java:360) [rhq-server-metrics-4.9.0.JON320GA-redhat-1.jar:4.9.0.JON320GA-redhat-1] at org.rhq.server.metrics.MetricsServer.findDataForResource(MetricsServer.java:189) [rhq-server-metrics-4.9.0.JON320GA-redhat-1.jar:4.9.0.JON320GA-redhat-1] at org.rhq.enterprise.server.measurement.MeasurementDataManagerBean.findDataForResource(MeasurementDataManagerBean.java:844) [rhq-server.jar:4.9.0.JON320GA-redhat-2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_21] ...
The bug is still occurring due to metrics in the twenty_four_hour_metrics table having a value of Double.NaN. The changes introduced in 3.2.2 check for invalid metrics where max < avg or min > avg. We need to also check for and filter out those NaN metrics.
I have added logic to InvalidMetricsManager to treat aggregate metrics having a value of Double.NaN as invalid. I committed/pushed the fix to the release/jon3.2.x branch. commit hash: 28e6e45129b03
Moving to ON_QA as this is available for test in JON 3.2.3 ER01 build: http://jon01.mw.lab.eng.bos.redhat.com:8042/dist/release/jon/3.2.3.GA/8-14-14/
Verified on version : 3.2.0.GA Update 03 Build Number :bca1bc8:e19c43d Used cqlsh with help from jsanda to create invalid metrics. The server log has below for invalid metrics: 19:52:30,502 WARN [org.rhq.server.metrics.MetricsServer] (http-/0.0.0.0:7080-18) The six_hour_metrics metric AggregatedNumericMetric[scheduleId=10007, avg=1.9998139733333333E9, min=1.9999E9, max=2.070183936E9, timestamp=1408343400000] is invalid. It will be excluded from the results sent to the client and we will attempt to recompute the metric. 19:52:30,558 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (http-/0.0.0.0:7080-18) Adding InvalidMetric{type=6 hour, scheduleId=10007, timestamp=1408343400000, max=2.070183936E9, min=1.9999E9, avg=1.9998139733333333E9} to queue for processing 19:52:30,742 WARN [org.rhq.server.metrics.MetricsServer] (http-/0.0.0.0:7080-18) The six_hour_metrics metric AggregatedNumericMetric[scheduleId=10013, avg=1.1249015466666667E8, min=1.0682368E8, max=1.06827776E8, timestamp=1408343400000] is invalid. It will be excluded from the results sent to the client and we will attempt to recompute the metric. 19:52:30,742 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (http-/0.0.0.0:7080-18) Adding InvalidMetric{type=6 hour, scheduleId=10013, timestamp=1408343400000, max=1.06827776E8, min=1.0682368E8, avg=1.1249015466666667E8} to queue for processing ---------- The invalid metric then recalculated and reflected in cqlsh query and UI. The server log has below: 20:03:47,864 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) Attempting to fix InvalidMetric{type=6 hour, scheduleId=10007, timestamp=1408343400000, max=2.070183936E9, min=1.9999E9, avg=1.9998139733333333E9}. This may include updates to 1 hour, 6 hour, and 24 hour metrics. 20:03:48,051 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) The invalid 6 hour metric AggregatedNumericMetric[scheduleId=10007, avg=1.9998139733333333E9, min=1.9999E9, max=2.070183936E9, timestamp=1408343400000] has been recomputed with a new value of {max: 2.070183936E9, min: 1.884680192E9, avg: 1.9998139733333333E9} 20:03:48,054 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) InvalidMetric{type=6 hour, scheduleId=10007, timestamp=1408343400000, max=2.070183936E9, min=1.9999E9, avg=1.9998139733333333E9} has been recomputed with a new value of {max: 2.27313664E9, min: 1.884680192E9, avg: 2.0850711665777779E9} 20:03:48,055 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) Attempting to fix InvalidMetric{type=6 hour, scheduleId=10013, timestamp=1408343400000, max=1.06827776E8, min=1.0682368E8, avg=1.1249015466666667E8}. This may include updates to 1 hour, 6 hour, and 24 hour metrics. 20:03:48,095 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) The invalid 6 hour metric AggregatedNumericMetric[scheduleId=10013, avg=1.1249015466666667E8, min=1.0682368E8, max=1.06827776E8, timestamp=1408343400000] has been recomputed with a new value of {max: 1.32644864E8, min: 1.0682368E8, avg: 1.1249015466666667E8} 20:03:48,097 INFO [org.rhq.server.metrics.invalid.InvalidMetricsManager] (pool-20-thread-1) InvalidMetric{type=6 hour, scheduleId=10013, timestamp=1408343400000, max=1.06827776E8, min=1.0682368E8, avg=1.1249015466666667E8} has been recomputed with a new value of {max: 4.502528E8, min: 1.0682368E8, avg: 2.3468955875555557E8}