Bug 1299549 - Performance degradation when using REST API on WebSphere
Performance degradation when using REST API on WebSphere
Status: VERIFIED
Product: JBoss BPMS Platform 6
Classification: JBoss
Component: Business Central (Show other bugs)
6.3.0
Unspecified Unspecified
urgent Severity urgent
: ER3
: 6.3.0
Assigned To: Shelly McGowan
Lukáš Petrovický
: TestBlocker
Depends On:
Blocks: 1310739
  Show dependency treegraph
 
Reported: 2016-01-18 10:32 EST by Tomas Livora
Modified: 2018-01-30 23:45 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1310739 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Tomas Livora 2016-01-18 10:32:00 EST
Description of problem:
REST API calls are becoming significantly slower over time when Business Central is run on WebSphere.

Version-Release number of selected component (if applicable):
6.2.0 GA

Steps to Reproduce:
1. Run WasRestPerformanceTest from jBPM Integration test suite [1]
2. See how test duration time is raising.

Actual results:
The same process run many times is becoming slower and slower. Here are the times to run 100 processes (in ms):
9937
20448
23608
29905
45294
60033

Expected results:
The execution should always take almost the same amount of time.

Additional info:
This is reproducible using both direct REST calls and provided remote client classes. The issue affects only REST interface. When you run the same processes using SOAP or JMS, you will see that there is no performance degradation over time.


[1] https://gitlab.mw.lab.eng.bos.redhat.com/bxms/brms/blob/master/test-jbpm-integration/src/test/java/org/jboss/qa/bpms/jbpm/integration/remote/WasRestPerformanceTest.java
Comment 1 Tomas Livora 2016-02-11 05:48:39 EST
This issue causes many problems when running tests from our test suite. The tests run 3-4 hours instead of 20 minutes on other containers. And there are also many failures caused by timeouts.
Comment 2 Marco Rietveld 2016-02-29 11:22:20 EST
Tomas, would it be possible for you to take a snapshot of the heap? That way I can at least look at it with MAT or another tool to analyze the memory used.
Comment 4 Marco Rietveld 2016-03-14 06:51:56 EDT
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1310739 for status updates.
Comment 5 Kris Verlaenen 2016-03-15 10:16:24 EDT
Tomas, is this a regression?  If so, do you know what the last version was that was working as expected?
Comment 6 Tomas Livora 2016-03-16 03:29:53 EDT
Kris, we discovered this problem on version 6.2. However, it might have been there for a while. The tests on WebSphere have always been a little bit slower than on other containers. But then I wrote extensive REST Query API tests, around 200 simple tests which run together only something more than 3 minutes on most containers and it started to take several hours to run the whole test suite on WebSphere. That was the moment when I noticed something is wrong there and started investigating it.
Comment 7 Marco Rietveld 2016-03-22 10:04:19 EDT
FYI: at the moment, I've been able to conclude the following: 

The problem is *not* a memory leak: if I let a test suite run (for more than 2 hours, roughly 20x), it plateaus at around 11x slower (~700 seconds as opposed to 60)  -- and (heap) dumps taken at the end show that the memory usage does *not* increase and otherwise remains steady at roughly 2/3ds of the heap. The reserved memory space (heap) also does not expand. 

I'm currently trying to isolate the problem by taking many "thread" (javacore) stack dumps during the test cycle (both at the beginning and end) and thus be able to find any unusual activity. 

I will probably/may also end up manually debuging a REST call on the server.
Comment 8 Marco Rietveld 2016-04-06 08:10:51 EDT
Fixed on 6.4.x. Commits: 

6.4.x: 
https://github.com/droolsjbpm/droolsjbpm-integration/commit/e8c4332f

Pull request for 6.3.x: 
https://github.com/droolsjbpm/droolsjbpm-integration/pull/403
Comment 9 Marco Rietveld 2016-04-11 09:19:59 EDT
https://bugzilla.redhat.com/show_bug.cgi?id=1299549
Comment 10 Marco Rietveld 2016-04-11 09:20:11 EDT
Unfortunately, while the performance problem is fixed, it looks like custom-type serialization is now broken. 

Currently working on fixing that..
Comment 11 Marco Rietveld 2016-04-11 09:35:34 EDT
(custom-type serialization is only broken on websphere, btw).
Comment 12 Tomas Livora 2016-04-12 05:05:45 EDT
I'm setting this back to MODIFIED since the fix has already been included in the ER3 tag. The problem with custom type serialization on WebSphere, which was found on community snapshot after the fix for this BZ was provided, will be tracked in another BZ when ER3 is released.
Comment 13 Marco Rietveld 2016-04-13 10:07:22 EDT
The above problem (custom-type serialization problems after the fix) can be fixed by changing the *configuration* of the websphere instance. 

In the custom properties of the JVM instance (reachable via the server configuration), add the following custom property: 

Property name:  org.apache.wink.jaxbcontextcache
Property value: off

This fixes the custom-type serialization problem. 

Also, obviously, it turns off caching for the JAXBContext instances. This makes performance a little bit slower, but not significantly -- especially since for the kie-remote-* services, caching isn't needed in the first place. Furthermore, it's also not needed for the kie-server REST API nor the guvnor REST API.
Comment 14 Tomas Livora 2016-04-18 08:46:55 EDT
Verified on BPM Suite 6.3.0 ER3

Note You need to log in before you can comment on or make changes to this bug.