Bug 739917 - Suspicious usage of ThreadPoolExecutor - possible performance problem
Summary: Suspicious usage of ThreadPoolExecutor - possible performance problem
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-java
Version: 2.0
Hardware: All
OS: All
medium
urgent
Target Milestone: ---
: ---
Assignee: messaging-bugs
QA Contact: MRG Quality Engineering
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-09-20 12:35 UTC by Martin Vecera
Modified: 2018-02-06 16:40 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:

Attachments (Terms of Use)
CPU usage all threads (725.94 KB, image/jpeg)
2011-09-20 12:35 UTC, Martin Vecera 		no flags Details
CPU usage waiting threads (628.30 KB, image/jpeg)
2011-09-20 12:35 UTC, Martin Vecera 		no flags Details
CPU usage blocked threads (498.38 KB, image/jpeg)
2011-09-20 12:36 UTC, Martin Vecera 		no flags Details
Threads activity (779.71 KB, image/jpeg)
2011-09-20 12:36 UTC, Martin Vecera 		no flags Details
Thread locks (144.21 KB, image/jpeg)
2011-09-20 12:37 UTC, Martin Vecera 		no flags Details
jProfiler 6.2.2 snapshot (2.83 MB, application/octet-stream)
2011-09-20 12:38 UTC, Martin Vecera 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Description Martin Vecera 2011-09-20 12:35:22 UTC 
Created attachment 524023 [details]
CPU usage all threads

I realized that message delivery and consumption takes inappropriate time while EAP server's HW and MRG server's HW show only low utilization (EAP ~10% CPU usage, MRG ~5% CPU usage and ~22% IO wait).

I run a test (for steps to run the test see bug no. 739898) with jProfiler installed and realized that ThreadPoolExecutor and its workers are blocked or waiting almost all the time. My suspicion is that there is a synchronization lock between individual threads preventing them to process messages in parallel.

Attached is jProfiler's snapshot and screenshots from it.

Unfortunatelly, given my schedule on the project, I'm not able to go any further in the analysis unless I have extra time.
Comment 1 Martin Vecera 2011-09-20 12:35:59 UTC 
Created attachment 524024 [details]
CPU usage waiting threads
Comment 2 Martin Vecera 2011-09-20 12:36:25 UTC 
Created attachment 524025 [details]
CPU usage blocked threads
Comment 3 Martin Vecera 2011-09-20 12:36:50 UTC 
Created attachment 524026 [details]
Threads activity
Comment 4 Martin Vecera 2011-09-20 12:37:20 UTC 
Created attachment 524027 [details]
Thread locks
Comment 5 Martin Vecera 2011-09-20 12:38:22 UTC 
Created attachment 524028 [details]
jProfiler 6.2.2 snapshot
Comment 6 Weston M. Price 2011-09-21 20:12:35 UTC 
This would be a JMS client issue as the JCA adapter does not control or influence in anyway the behavior of IoReceiver and it's associated classes. Further, we do not pool JMS sessions on inbound delivery so a synchronization issue would not be in the adapter itself.  As a result, I am changing the component of this bug to quid-java.

I am also changing the target milestone as this is going to require more analysis and work that the 2.0.4 time frame allows at this point.
Comment 7 Martin Vecera 2012-01-09 15:58:16 UTC 
Latest performance results are here - https://docspace.corp.redhat.com/docs/DOC-88691

Now I work on freash jProfiler report. PM should decide whether these numbers are sufficient, however, it they do not comply with our PRD.
Note You need to log in before you can comment on or make changes to this bug.