Bug 1121027

Summary: JobExecutor: EntityManager closed error on join tx.
Product: [Retired] JBoss BPMS Platform 6 Reporter: Marek Baluch <mbaluch>
Component: jBPM CoreAssignee: Alessandro Lazarotti <alazarot>
Status: CLOSED EOL QA Contact: Marek Baluch <mbaluch>
Severity: medium Docs Contact:
Priority: high    
Version: 6.1.0CC: mbaluch
Target Milestone: ER3   
Target Release: 6.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-27 20:00:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marek Baluch 2014-07-18 08:23:45 UTC
*** Description of problem:
Sometimes when the job executor uses more threads (e.g. 4) to execute queued jobs the following exception may occur.

java.lang.IllegalStateException: EntityManager is closed
    at org.hibernate.ejb.AbstractEntityManagerImpl.joinTransaction(AbstractEntityManagerImpl.java:1181) ~[hibernate-entitymanager-4.2.12.Final-redhat-1.jar:4.2.12.Final-redhat-1]
    at org.jbpm.shared.services.impl.JpaPersistenceContext.joinTransaction(JpaPersistenceContext.java:169) ~[jbpm-shared-services-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
    at org.jbpm.shared.services.impl.TransactionalCommandService.execute(TransactionalCommandService.java:37) ~[jbpm-shared-services-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
    at org.jbpm.executor.impl.jpa.ExecutorQueryServiceImpl.getRequestForProcessing(ExecutorQueryServiceImpl.java:196) ~[jbpm-executor-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
    at org.jbpm.executor.impl.AvailableJobsExecutor.executeJob(AvailableJobsExecutor.java:89) ~[jbpm-executor-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
    at org.jbpm.executor.impl.ExecutorRunnable.run(ExecutorRunnable.java:42) [jbpm-executor-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_51]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_51]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

*** How reproducible
approximately 20%

*** How to reproduce
Setup the job executor to use 4 threads and sleep interval 2 seconds. Create a simple
process with one async job (I used the Async Data Executor from async samples).

Start the process and Thread.sleep() for approximately 5-6 seconds so there's time for
the job executor to pick up and execute the task.

Comment 1 Maciej Swiderski 2014-07-23 17:47:27 UTC
Marek,

do you run this as unit test or in kie-wb/business-central?

If unit test then would be great to get reproducer so I can figure out why entity manager is wrongly used.

Comment 2 Marek Baluch 2014-07-23 18:05:07 UTC
Maciej,

it's a unit test. I will create the reproducer and get back to you.

Comment 3 Marek Baluch 2014-08-04 09:45:56 UTC
Maciej,

a test case can be found here:

git+ssh://code.engineering.redhat.com/jbossqe/brms.git

it's AsyncTaskTest located in project test-jbpm-regression.

Please note that the issue is really hard to reproduce. I had most luck when I used 

    private static final int EXECUTOR_THREADS = 4;
    private static final int EXECUTOR_INTERVAL = 2;

The above fields are located in JbpmAsyncJobTetsCase which is the class extended by AsyncTaskTest.

Comment 4 Marek Baluch 2014-08-04 09:48:57 UTC
One more issue I see when I execute the AsyncTaskTest#testTaskComplete() is the following.

[2014-08-04 11:37:28,405] WARN - Error during command com.bpms.functional.jobexec.UserCommand execution There is no runtime manager for deployment default-singleton

This causes the job to be executed twice when it should be just once. This is a bit better reproducible than the above issue.

My apologies but I don't have a straight way how to reproduce this yet. The parallelism makes it really hard.

Comment 5 Maciej Swiderski 2014-08-08 12:43:21 UTC
Marek, the second issue is sort of expected as I can see you disposing runtime manager during the test while executor is still running so it might end up to execute job in between dispose and create of runtime manager and thus not being able to find it. So that's what I would expect to happen.

When it comes to the first issue, I tried hard to reproduce it but couldn't. It might be because it was already enhanced as I did some improvements while working on the ejb services. So might be worth to retest that once you get build with them.

Another thing is that not sure disposing and recreating runtime manager makes sense in this tests as it might cause more issues then good. Just my two cents about it...

Comment 6 Marek Baluch 2014-08-13 09:48:56 UTC
(In reply to Maciej Swiderski from comment #5)
> Marek, the second issue is sort of expected as I can see you disposing
> runtime manager during the test while executor is still running so it might
> end up to execute job in between dispose and create of runtime manager and
> thus not being able to find it. So that's what I would expect to happen.

yes - this is a copy/paste error from another test. Thank you for pointing that out.

> When it comes to the first issue, I tried hard to reproduce it but couldn't.
> It might be because it was already enhanced as I did some improvements while
> working on the ejb services. So might be worth to retest that once you get
> build with them.

yes - I will retry.

> Another thing is that not sure disposing and recreating runtime manager
> makes sense in this tests as it might cause more issues then good. Just my
> two cents about it...

You got me at a disadvantage here. Why would it do more harm than good (as long as the manager is properly disposed)? Would you be so kind and explain this a bit in more detail?

Thank you.

Comment 7 Maciej Swiderski 2014-08-27 05:35:41 UTC
Marek,

what I meant is that runtime manager close can be seen as undeployment and undeployment should not be allowed with active process instances. Although this is not secured when using RuntimeManager directly but only when using jbpm services. The problem with such scenario is that there is executor service still running and it has jobs to be executed for given runtime manager but if you remove that runtime manager there is no way to continue. So either to skip such operation (recreate of runtime manager) or do recreation of executor service as well to sort of simulate complete environment restart.
Does that make sense?

Comment 8 Marek Baluch 2014-08-27 11:53:14 UTC
Yes - now it makes perfect sense. Thanks!

As for the issue it self. I suggest to keep it open for the time being and lower the priority. That should make sure it will not show up on the radar. We'll see if it pops up again or not.

Comment 9 Maciej Swiderski 2014-11-17 12:14:54 UTC
Marek, shall we mark it as modified so it gets retested?

Comment 10 Marek Baluch 2014-11-20 07:11:39 UTC
Yes - I hope I will not stumble upon this anymore.

Comment 11 Marek Baluch 2015-01-03 19:44:22 UTC
Verified on 6.1.0.ER3.

The issue was seen but only as a WARN message and tests do not fail anymore because of it.