Bug 1089838

Summary: [GSS] (6.3.0) Full scheduledReferences traversal in every call to ScheduledDeliveryHandlerImpl$ScheduledDeliveryRunnable.run()
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Jimmy Wilson <jawilson>
Component: HornetQAssignee: Clebert Suconic <csuconic>
Status: CLOSED CURRENTRELEASE QA Contact: Miroslav Novak <mnovak>
Severity: unspecified Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.2.1CC: kkhan, msvehla, smumford, vtunka
Target Milestone: DR2   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
In previous versions of JBoss EAP 6, if a very large number of messages were scheduled with a small interval, excessive CPU load would result. The root cause of this issue was that at every instance messages were to be consumed, the *whole* of the `scheduledReferences` linked list was traversed unecessarily. This issue has been resolved in this release.
 Story Points: ---
Clone Of: 1088600 Environment:
Last Closed: 2014-08-06 14:36:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1064596, 1084639, 1088600    

Description Jimmy Wilson 2014-04-22 04:34:37 UTC 
+++ This bug was initially created as a clone of Bug #1088600 +++

The method ScheduledDeliveryHandlerImpl$ScheduledDeliveryRunnable.run() unnecessarily does a full traversal of the scheduledReferences linked list on each and every call.

When the number of scheduled messages is very large, this traversal severely impacts HornetQ's performance when consuming scheduled messages.

We are using HornetQ 2.2.24 in an application where we schedule ~400,000 messages in a single queue to be consumed several hours into the future, with large bursts of messages to be consumed as quickly as possible. In this application, we have found that HornetQ's rate of consuming scheduled messages is limited by CPU. From performance profiling, we found that the method

org.hornetq.core.server.impl.ScheduledDeliveryHandlerImpl$ScheduledDeliveryRunnable.run()

is consuming approximately 70-80% of the CPU cycles during bursts. Looking at the code, this is due to the full traversal of the scheduledReferences linked list on every call to run():
https://github.com/hornetq/hornetq/blob/HornetQ_2_2_24_EAP_GA/src/main/org/hornetq/core/server/impl/ScheduledDeliveryHandlerImpl.java#L181
Comment 1 Kabir Khan 2014-07-01 14:26:41 UTC 
Should be fixed by upgrade https://bugzilla.redhat.com/show_bug.cgi?id=1064596 for DR2
Comment 2 Miroslav Novak 2014-07-02 07:57:41 UTC 
Verified using provided ScheduledDeliveryHandlerTest during EAP 6.3.0.ER7. Thanks!