Bug 1648762

Summary: [GSS](6.4.z) Lock is not released when JTS is enabled and a timer is cancelled inside a transaction
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Lyle Wang <lywang>
Component: EJBAssignee: Radovan Netuka <rnetuka>
Status: CLOSED CURRENTRELEASE QA Contact: Peter Mackay <pmackay>
Severity: high Docs Contact:
Priority: high    
Version: 6.4.18CC: bmaxwell, david.lloyd, iweiss, jbaesner, pmackay, rnetuka, sappleto, thofman
Target Milestone: CR1   
Target Release: EAP 6.4.22   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-19 12:41:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1613417, 1653367, 1653825    
Attachments:
Description Flags
patch none

Description Lyle Wang 2018-11-12 04:45:43 UTC
Description of problem:

Lock is held by EJB timer (TimerServiceImpl)  and in case JTS transaction is cancelled the lock won't be released correctly . This code is the problem:
https://github.com/wildfly/wildfly/blob/78dee79dd0f49c6cbd2b8db5d8640980faa08892/ejb3/src/main/java/org/jboss/as/ejb3/timerservice/TimerServiceImpl.java#L640

Basically it holds the timer lock until the transaction completes, and then attempts to release it in afterCompletion. The problem is that when JTS is enabled afterCompletion will be called by a seperate thread, which can't call unlock as it is not the owner.

A simple fix could be to just change the lock to a semaphore, so that the other thread can release it.




Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Start a JTA/JTS tx and call an EJB timer inside it
2. Make the transaction timeout
3. Capture a thread dump and see a thread like below  (0a0818b80 is locked object)
---------------------------------------------------------------------------------------------------------
"Incoming-2,RouterPoliciesClusterGroup,svc-3-comecimpolicy-59228 (, payload=52 bytes)" #661 prio=5 os_prio=0 tid=0x00007f5214c2d000 nid=0x1e67 waiting on condition [0x00007f520ac97000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000a0818b80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
    - <0x00000000ea1f4c48> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
---------------------------------------------------------------------------------------------------------


Actual results:
Lock being held forever

Expected results:
Lock getting released


Additional info:

Comment 5 Brad Maxwell 2018-11-26 18:16:09 UTC
Created attachment 1508629 [details]
patch

Comment 7 Peter Mackay 2019-05-08 20:27:06 UTC
Regression tests passed
Verified with EAP 6.4.22.CP.CR1