Bug 777577 (SOA-85)

Summary: ESB performance bug blocking large queue backlog stress testing
Product: [JBoss] JBoss Enterprise SOA Platform 4 Reporter: Len DiMaggio <ldimaggi>
Component: JBossESBAssignee: Kevin Conner <kevin.conner>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.2 IR3CC: ldimaggi, rcampbell
Target Milestone: ---   
Target Release: 4.2 IR9   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/SOA-85
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
RHEL5, Java 1.5 , ESB 4.2.1 IR3
Last Closed: 2008-01-07 12:03:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Len DiMaggio 2007-11-01 15:48:28 UTC
Date of First Response: 2007-11-20 08:39:02
project_key: SOA

Problem description:

This bug blocks us from running ESB stress tests for longer than 20 minutes. 

The bug was originally reported in JBESB-880. If between 10 and 100 messages per second generated, the ESB server becomes unresponsive and the consoles are locked. If the traffic stream is stopped, the message queue may be process and the server may recover, but sometimes it hangs indefinitely.

This condition - a large queue backlog - can happen if an ESB app has an outage. Or, is a client app sends messages faster then ESB reads them from queue - for example, a batch process - such as a job that starts up at midnight and generates a large number of messages

Test description:

The test exercises clear throughput of the ESB - no influence of alien code - the server deployed is a no-op/placeholder. The test:

1) Generates big amount of messages to the JMS GW queue
2) Deploys the ESB application
3) When the first message is received the internal message counter and timer is started
4) When the last message is processed the speed is calculated and test is ended

Steps to run test: 

1) Obtain the ESB bits 
2) Obtain dbinstall (see below)
3) Setup and run dbinstall to have custom ESB config for database
4) Obtain quickstart-based tests from https://svn.corp.jboss.com/repos/soa/trunk/qa/tests/
5) Setup build.properties in tests/quickstart/tests and point it on ESB bits and created db configuration
6) Run ant rebuild-ng - it will build framework, tests and setup ESB
7) Run the test with 10 messages
8) If the test is successful, rerun with larger numbers of messages

You can also check the script that is used for hudson matrix build project jboss-soa-platform-IntegrationTest-testsuite-matrix - it performs the same setup steps)

Test source:
https://svn.corp.jboss.com/repos/soa/trunk/qa/tests/quickstarts/performance1

Documentation:
https://svn.corp.jboss.com/repos/soa/trunk/qa/tests/quickstarts/tests/performance_tests.txt

DB setup scripts:
https://svn.corp.jboss.com/repos/soa/trunk/qa/dbinstall/ - just edit the build.properties and run ant

(Test developed by Jirka Pechanec - thanks Jirka!)

Comment 1 Len DiMaggio 2007-11-01 15:49:00 UTC
Link: Added: This issue depends JBESB-880


Comment 2 Len DiMaggio 2007-11-01 15:49:18 UTC
Link: Added: This issue is related to JBQA-1029


Comment 3 Len DiMaggio 2007-11-01 15:51:11 UTC
Link: Added: This issue is a dependency of SOA-84


Comment 4 Len DiMaggio 2007-11-01 15:51:30 UTC
Link: Removed: This issue is a dependency of SOA-84 


Comment 5 Len DiMaggio 2007-11-01 15:52:05 UTC
Link: Added: This issue related SOA-84


Comment 7 Len DiMaggio 2007-11-05 19:23:15 UTC
Link: Added: This issue related SOA-93


Comment 10 Len DiMaggio 2007-11-08 19:08:41 UTC
Link: Removed: This issue related SOA-84 


Comment 11 Len DiMaggio 2007-11-08 19:09:10 UTC
Link: Added: This issue related SOA-84


Comment 12 Mark Little 2007-11-20 13:39:02 UTC
Kevin (paraphrasing) "on going, work progressing and checking some other performance issues."

Comment 13 Jiri Pechanec 2007-12-11 12:02:50 UTC
Link: Removed: This issue related SOA-84 


Comment 14 trev 2007-12-17 17:12:17 UTC
fixes merged in from ESB head - revision 17282, if performance needs revisiting please reopen against beta1

Comment 16 Martin Vecera 2008-01-04 07:21:25 UTC
I run Helloworld performance test in optimized sending mode over night. The server was able to process 1,000,000 messages within 31546843ms = 8.76 hours. This makes average throughput 31.7 messages per second (unfortunately I left file log level to DEBUG and ended with >1GB log file, I have to rerun it). After that the server was perfectly responsive, no errors appeared on the output and the server was able to gracefully terminate.

Comment 17 Martin Vecera 2008-01-04 08:16:29 UTC
The speed with file log level set to WARN is definitely better (~50 msgs/s).


Comment 18 Martin Vecera 2008-01-07 12:03:24 UTC
Verified.

Comment 19 nwallace 2008-09-26 06:42:21 UTC
Link: Added: This issue related SOA-877