Bug 1019306 - Never ending exception loop when shutting down executor thread which runs many processes with timers
Never ending exception loop when shutting down executor thread which runs man...
Status: CLOSED NOTABUG
Product: JBoss BPMS Platform 6
Classification: JBoss
Component: jBPM Core (Show other bugs)
6.0.0
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Maciej Swiderski
Jiri Svitak
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-15 09:02 EDT by Jiri Svitak
Modified: 2015-06-01 21:35 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-18 09:45:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
perfcake.log (239.21 KB, text/plain)
2013-10-15 09:02 EDT, Jiri Svitak
no flags Details

  None (edit)
Description Jiri Svitak 2013-10-15 09:02:09 EDT
Created attachment 812508 [details]
perfcake.log

Never ending exception loop when shutting down executor thread which runs many processes with timers. I have found this using process flood project for performance testing. PerfCake framework uses executor threads for sender, in this case just one thread was enough. Measurement is fine. When the PerfCake wants to stop measurement (stop executor thread), then never ending loop of transaction errors and warns occurs. Whole log is attached.

2013-10-15 14:44:53,418 WARN  [org.drools.persistence.jta.JtaTransactionManager] Unable to begin transaction
bitronix.tm.internal.BitronixSystemException: cannot start a new transaction, transaction manager is shutting down
...
2013-10-15 14:44:53,419 WARN  [org.drools.persistence.SingleSessionCommandService] Could not commit session
java.lang.RuntimeException: Unable to begin transaction
...
2013-10-15 14:44:53,420 WARN  [org.drools.persistence.jta.JtaTransactionManager] Unable to rollback transaction
java.lang.IllegalStateException: no transaction started on this thread
...
2013-10-15 14:44:53,420 ERROR [org.drools.persistence.SingleSessionCommandService] Could not rollback
java.lang.RuntimeException: Unable to rollback transaction
...


To reproduce:
1.) git clone git://git.app.eng.bos.redhat.com/jbossqe-process-flood.git
2.) cd jbossqe-process-flood/bpms-6.x
3.) ./execute_scenario.sh -Dscenario=Timer_Java -s ../mysql-localhost.xml
Make sure that you have empty MySQL database accessible via properties defined in mysql-localhost.xml file.
4.) Never ending loop runs. Shut it down with Ctrl+C and check the log file:
less perfcake.log

My environment:
BPMS 6 ER4 (6.0.0-redhat-4)
PerfCake 1.0 (available in maven central)
MySQL 5.5.32
Singleton runtime manager, single thread
Comment 1 Maciej Swiderski 2013-10-17 07:39:31 EDT
the errors are throw because runtime manager used to start process instances was not closed and since there are timers these timers start firing after transaction manager has been shutdown and thus errors are seen.

Please replace close method of AbstractProcessSender with following:
    @Override
    public void close() {
        ((AbstractRuntimeManager)runtimeManager).close(true);
        SessionManagementUtil.close();
    }
that will resolve the issue with never ending exceptions on shutdown of the test. Casting to AbstractRuntimeManager is required to be able to use close(boolean) method that allows to completely remove timer jobs on close.
Comment 2 Jiri Svitak 2013-10-18 07:15:03 EDT
Thanks, it worked.

However I think that such method close(boolean) should be part of RuntimeManager interface (kie api), as this situation will occur often. AbstractRuntimeManager is an internal impl class and developer should not be forced to use internal api.
What do you think?
Comment 3 Maciej Swiderski 2013-10-18 07:21:02 EDT
not really, the internal part of it is there only for undeployment functionality and if you use jbpm-kie-services then you operate on deployment service level. In usual cases close will be enough for most of the cases as you don't want to destroy it but just close it on runtime.
I suggested you to use this internal version of close method as it will ensure that none of the timers will fire during shutdown as you don't need them any more so they can be freely removed. In most cases when you close runtime manager you don't won't destroy the data it produced (timer jobs) as you would most likely would like to start it later on.
Comment 4 Jiri Svitak 2013-10-18 09:45:53 EDT
Ok.

Closing the BZ.

Note You need to log in before you can comment on or make changes to this bug.