Bug 739917
| Summary: | Suspicious usage of ThreadPoolExecutor - possible performance problem | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Martin Vecera <mvecera> | ||||||||||||||
| Component: | qpid-java | Assignee: | messaging-bugs <messaging-bugs> | ||||||||||||||
| Status: | NEW --- | QA Contact: | MRG Quality Engineering <mrgqe-bugs> | ||||||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||||||
| Priority: | medium | ||||||||||||||||
| Version: | 2.0 | CC: | jross | ||||||||||||||
| Target Milestone: | --- | ||||||||||||||||
| Target Release: | --- | ||||||||||||||||
| Hardware: | All | ||||||||||||||||
| OS: | All | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||
| Last Closed: | Type: | --- | |||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||
| Embargoed: | |||||||||||||||||
| Attachments: |
|
||||||||||||||||
Created attachment 524024 [details]
CPU usage waiting threads
Created attachment 524025 [details]
CPU usage blocked threads
Created attachment 524026 [details]
Threads activity
Created attachment 524027 [details]
Thread locks
Created attachment 524028 [details]
jProfiler 6.2.2 snapshot
This would be a JMS client issue as the JCA adapter does not control or influence in anyway the behavior of IoReceiver and it's associated classes. Further, we do not pool JMS sessions on inbound delivery so a synchronization issue would not be in the adapter itself. As a result, I am changing the component of this bug to quid-java. I am also changing the target milestone as this is going to require more analysis and work that the 2.0.4 time frame allows at this point. Latest performance results are here - https://docspace.corp.redhat.com/docs/DOC-88691 Now I work on freash jProfiler report. PM should decide whether these numbers are sufficient, however, it they do not comply with our PRD. |
Created attachment 524023 [details] CPU usage all threads I realized that message delivery and consumption takes inappropriate time while EAP server's HW and MRG server's HW show only low utilization (EAP ~10% CPU usage, MRG ~5% CPU usage and ~22% IO wait). I run a test (for steps to run the test see bug no. 739898) with jProfiler installed and realized that ThreadPoolExecutor and its workers are blocked or waiting almost all the time. My suspicion is that there is a synchronization lock between individual threads preventing them to process messages in parallel. Attached is jProfiler's snapshot and screenshots from it. Unfortunatelly, given my schedule on the project, I'm not able to go any further in the analysis unless I have extra time.