*** Description of problem:
Sometimes when the job executor uses more threads (e.g. 4) to execute queued jobs the following exception may occur.
java.lang.IllegalStateException: EntityManager is closed
at org.hibernate.ejb.AbstractEntityManagerImpl.joinTransaction(AbstractEntityManagerImpl.java:1181) ~[hibernate-entitymanager-4.2.12.Final-redhat-1.jar:4.2.12.Final-redhat-1]
at org.jbpm.shared.services.impl.JpaPersistenceContext.joinTransaction(JpaPersistenceContext.java:169) ~[jbpm-shared-services-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
at org.jbpm.shared.services.impl.TransactionalCommandService.execute(TransactionalCommandService.java:37) ~[jbpm-shared-services-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
at org.jbpm.executor.impl.jpa.ExecutorQueryServiceImpl.getRequestForProcessing(ExecutorQueryServiceImpl.java:196) ~[jbpm-executor-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
at org.jbpm.executor.impl.AvailableJobsExecutor.executeJob(AvailableJobsExecutor.java:89) ~[jbpm-executor-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
at org.jbpm.executor.impl.ExecutorRunnable.run(ExecutorRunnable.java:42) [jbpm-executor-6.1.0.CR1-redhat-1.jar:6.1.0.CR1-redhat-1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_51]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_51]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
*** How reproducible
*** How to reproduce
Setup the job executor to use 4 threads and sleep interval 2 seconds. Create a simple
process with one async job (I used the Async Data Executor from async samples).
Start the process and Thread.sleep() for approximately 5-6 seconds so there's time for
the job executor to pick up and execute the task.
do you run this as unit test or in kie-wb/business-central?
If unit test then would be great to get reproducer so I can figure out why entity manager is wrongly used.
it's a unit test. I will create the reproducer and get back to you.
a test case can be found here:
it's AsyncTaskTest located in project test-jbpm-regression.
Please note that the issue is really hard to reproduce. I had most luck when I used
private static final int EXECUTOR_THREADS = 4;
private static final int EXECUTOR_INTERVAL = 2;
The above fields are located in JbpmAsyncJobTetsCase which is the class extended by AsyncTaskTest.
One more issue I see when I execute the AsyncTaskTest#testTaskComplete() is the following.
[2014-08-04 11:37:28,405] WARN - Error during command com.bpms.functional.jobexec.UserCommand execution There is no runtime manager for deployment default-singleton
This causes the job to be executed twice when it should be just once. This is a bit better reproducible than the above issue.
My apologies but I don't have a straight way how to reproduce this yet. The parallelism makes it really hard.
Marek, the second issue is sort of expected as I can see you disposing runtime manager during the test while executor is still running so it might end up to execute job in between dispose and create of runtime manager and thus not being able to find it. So that's what I would expect to happen.
When it comes to the first issue, I tried hard to reproduce it but couldn't. It might be because it was already enhanced as I did some improvements while working on the ejb services. So might be worth to retest that once you get build with them.
Another thing is that not sure disposing and recreating runtime manager makes sense in this tests as it might cause more issues then good. Just my two cents about it...
(In reply to Maciej Swiderski from comment #5)
> Marek, the second issue is sort of expected as I can see you disposing
> runtime manager during the test while executor is still running so it might
> end up to execute job in between dispose and create of runtime manager and
> thus not being able to find it. So that's what I would expect to happen.
yes - this is a copy/paste error from another test. Thank you for pointing that out.
> When it comes to the first issue, I tried hard to reproduce it but couldn't.
> It might be because it was already enhanced as I did some improvements while
> working on the ejb services. So might be worth to retest that once you get
> build with them.
yes - I will retry.
> Another thing is that not sure disposing and recreating runtime manager
> makes sense in this tests as it might cause more issues then good. Just my
> two cents about it...
You got me at a disadvantage here. Why would it do more harm than good (as long as the manager is properly disposed)? Would you be so kind and explain this a bit in more detail?
what I meant is that runtime manager close can be seen as undeployment and undeployment should not be allowed with active process instances. Although this is not secured when using RuntimeManager directly but only when using jbpm services. The problem with such scenario is that there is executor service still running and it has jobs to be executed for given runtime manager but if you remove that runtime manager there is no way to continue. So either to skip such operation (recreate of runtime manager) or do recreation of executor service as well to sort of simulate complete environment restart.
Does that make sense?
Yes - now it makes perfect sense. Thanks!
As for the issue it self. I suggest to keep it open for the time being and lower the priority. That should make sure it will not show up on the radar. We'll see if it pops up again or not.
Marek, shall we mark it as modified so it gets retested?
Yes - I hope I will not stumble upon this anymore.
Verified on 6.1.0.ER3.
The issue was seen but only as a WARN message and tests do not fail anymore because of it.