Bug 1019306

Summary: Never ending exception loop when shutting down executor thread which runs many processes with timers
Product: [Retired] JBoss BPMS Platform 6 Reporter: Jiri Svitak <jsvitak>
Component: jBPM CoreAssignee: Maciej Swiderski <mswiders>
Status: CLOSED NOTABUG QA Contact: Jiri Svitak <jsvitak>
Severity: high Docs Contact:
Priority: high    
Version: 6.0.0CC: mbaluch, mswiders
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-18 13:45:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
perfcake.log none

Description Jiri Svitak 2013-10-15 13:02:09 UTC
Created attachment 812508 [details]
perfcake.log

Never ending exception loop when shutting down executor thread which runs many processes with timers. I have found this using process flood project for performance testing. PerfCake framework uses executor threads for sender, in this case just one thread was enough. Measurement is fine. When the PerfCake wants to stop measurement (stop executor thread), then never ending loop of transaction errors and warns occurs. Whole log is attached.

2013-10-15 14:44:53,418 WARN  [org.drools.persistence.jta.JtaTransactionManager] Unable to begin transaction
bitronix.tm.internal.BitronixSystemException: cannot start a new transaction, transaction manager is shutting down
...
2013-10-15 14:44:53,419 WARN  [org.drools.persistence.SingleSessionCommandService] Could not commit session
java.lang.RuntimeException: Unable to begin transaction
...
2013-10-15 14:44:53,420 WARN  [org.drools.persistence.jta.JtaTransactionManager] Unable to rollback transaction
java.lang.IllegalStateException: no transaction started on this thread
...
2013-10-15 14:44:53,420 ERROR [org.drools.persistence.SingleSessionCommandService] Could not rollback
java.lang.RuntimeException: Unable to rollback transaction
...


To reproduce:
1.) git clone git://git.app.eng.bos.redhat.com/jbossqe-process-flood.git
2.) cd jbossqe-process-flood/bpms-6.x
3.) ./execute_scenario.sh -Dscenario=Timer_Java -s ../mysql-localhost.xml
Make sure that you have empty MySQL database accessible via properties defined in mysql-localhost.xml file.
4.) Never ending loop runs. Shut it down with Ctrl+C and check the log file:
less perfcake.log

My environment:
BPMS 6 ER4 (6.0.0-redhat-4)
PerfCake 1.0 (available in maven central)
MySQL 5.5.32
Singleton runtime manager, single thread

Comment 1 Maciej Swiderski 2013-10-17 11:39:31 UTC
the errors are throw because runtime manager used to start process instances was not closed and since there are timers these timers start firing after transaction manager has been shutdown and thus errors are seen.

Please replace close method of AbstractProcessSender with following:
    @Override
    public void close() {
        ((AbstractRuntimeManager)runtimeManager).close(true);
        SessionManagementUtil.close();
    }
that will resolve the issue with never ending exceptions on shutdown of the test. Casting to AbstractRuntimeManager is required to be able to use close(boolean) method that allows to completely remove timer jobs on close.

Comment 2 Jiri Svitak 2013-10-18 11:15:03 UTC
Thanks, it worked.

However I think that such method close(boolean) should be part of RuntimeManager interface (kie api), as this situation will occur often. AbstractRuntimeManager is an internal impl class and developer should not be forced to use internal api.
What do you think?

Comment 3 Maciej Swiderski 2013-10-18 11:21:02 UTC
not really, the internal part of it is there only for undeployment functionality and if you use jbpm-kie-services then you operate on deployment service level. In usual cases close will be enough for most of the cases as you don't want to destroy it but just close it on runtime.
I suggested you to use this internal version of close method as it will ensure that none of the timers will fire during shutdown as you don't need them any more so they can be freely removed. In most cases when you close runtime manager you don't won't destroy the data it produced (timer jobs) as you would most likely would like to start it later on.

Comment 4 Jiri Svitak 2013-10-18 13:45:53 UTC
Ok.

Closing the BZ.