Bug 1383572

Summary: [GSS](6.4.z) Deadlock in BasicAction when jboss remoting and JTA is used
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Flavia Rainone <frainone>
Component: EJBAssignee: Radovan Netuka <rnetuka>
Status: CLOSED NEXTRELEASE QA Contact: Jan Martiska <jmartisk>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.10CC: bmaxwell, david.lloyd, frainone, jawilson, jbilek, jtruhlar, ochaloup, ppalaga, ppenicka, rnetuka, sfikes, tom.ross
Target Milestone: CR1   
Target Release: EAP 6.4.12   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: eap6412-proposed
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1399703 (view as bug list) Environment:
Last Closed: 2016-11-29 15:39:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1238925, 1375585, 1383246, 1399703    
Attachments:
Description Flags
client application that reproduces the bug (requires db provisioning) none

Description Flavia Rainone 2016-10-11 06:09:38 UTC
At EJBRemoteTransactionPropagatingInterceptor, a transaction that has been already open at current server could be ignored, thus causing the transaction to be reimported, resulting in the double diamond problem.

To reproduce you need a reinvocation at a server that was already participating in the transaction, as the scenario below for example:
    server1 -> server2 -> server1

Comment 1 JBoss JIRA Server 2016-10-14 09:34:10 UTC
Flavia Rainone <frainone> updated the status of jira JBEAP-6193 to Coding In Progress

Comment 2 JBoss JIRA Server 2016-10-15 09:37:42 UTC
Flavia Rainone <frainone> updated the status of jira JBEAP-6394 to Coding In Progress

Comment 5 Peter Palaga 2016-10-19 08:19:57 UTC
Switching back to POST, because https://github.com/jbossas/jboss-eap/pull/2860 is not merged yet. I switched to MODIFIED because I saw the "Deadlock in the JTA remoting" commit merged in https://github.com/jbosstm/jboss-transaction-spi/commits/7.1.0.SPx . This is a reason why it is impractical to have one BZ to track changes in more than one git repo.

Comment 7 Jiří Bílek 2016-11-14 11:41:49 UTC
The fix works well on EAP 7.1.0 (our integration test for this issue does not work on 7.1.0.DR6 and works well on 7.1.0.DR7).
But for 6.4.12 it seems bad. EAP 6.4.12 gets the same issue as 6.4.11 and 7.1.0.DR6. The test looks right but for sure I would like to reproduce it with some reproducer.
Flavia could you provide some reproducer that is possible to run on EAP 6.4.x ?

Comment 8 Flavia Rainone 2016-11-22 12:42:17 UTC
Yes, the reproducer I used was provided by Tom Ross, based on the client's code. Woult that be enough?
This is the reproducer that I used to check if the fix was good.

Comment 9 Jiří Bílek 2016-11-22 13:04:17 UTC
Could you please attache the reproducer to this bz or send me it by en email?

Comment 10 Flavia Rainone 2016-11-22 14:49:40 UTC
I'll try to attach it to this bz, and I'll send an e-mail to Tom asking him to provision a database. The problem is that the reproducer is an app and requires a provisioned dataase, I'll send additional data by e-mail.

Comment 11 Flavia Rainone 2016-11-22 18:49:50 UTC
Created attachment 1222820 [details]
client application that reproduces the bug (requires db provisioning)

I'll explain in detail how to reproduce the bug once I have confirmed that we can have the db2 provisioned.

Comment 12 Jiří Bílek 2016-11-23 08:25:21 UTC
Thank you for reproducer. Provisioning of db2 is not necessary I can use our db2.

Comment 13 Flavia Rainone 2016-11-24 17:26:26 UTC
Okay, here are the instructions:
To reproduce  the issue follow the steps:
 - unzip the bundle.zip into, let’s say TEST_LOG directory 

 - build the test case project

      cd $TEST_LOC/bundle/test-deadlock
      mvn clean package


 - install JBoss EAP 6.4 CP9 
 - create two instance server-1 and server-2

      cd $JBOSS_HOME
      cp -rp standalone standalone-server-1
      cp -rp standalone standalone-server-2

      cp  $TEST_LOC/bundle/test-deadlock/eap-config/*.properties $JBOSS_HOME/standalone-server-1/configuration
      cp  $TEST_LOC/bundle/test-deadlock/eap-config/*.properties $JBOSS_HOME/standalone-server-2/configuration

      cp $TEST_LOC/bundle/test-deadlock/eap-config/server-1/standalone.xml to $JBOSS_HOME/standalone-server-1/configuration
      cp $TEST_LOC/bundle/test-deadlock/eap-config/server-2/standalone.xml to $JBOSS_HOME/standalone-server-2/configuration


 unzip the $TEST_LOC/bundle/db2.zip into $JBOSS_HOME/modules. db2.zip contains DB@ JDBC driver.

     cp $TEST_LOC/bundle/db2.zip $JBOSS_HOME/modules
     cd $JBOSS_HOME/module
     unzip db2.zip



start two JBoss instances

    cd $JBOSS_HOME/bin
     ./standalone.sh  -Djboss.server.base.dir=$JBOSS_HOME/standalone-server-1 -Djboss.node.name=server1


in another terminal session

    cd $JBOSS_HOME/bin
    ./standalone.sh  -Djboss.server.base.dir=$JBOSS_HOME/standalone-server-2 -Djboss.node.name=server2 -Djboss.socket.binding.port-offset=100


in another terminal

    cd $JBOSS_HOME/bin
    ./jboss-cli.sh —connect

    deploy $TEST_LOC/bundle/test-deadlock/project1-war/target/project1-war-0.0.1-SNAPSHOT.war


in yet another terminal session 

    cd $JBOSS_HOME/bin
    ./jboss-cli.sh —connect —controller=localhost:10099
    deploy $TEST_LOC/bundle/test-deadlock/project2-war/target/project2-war-0.0.1-SNAPSHOT.war


(I don’t think that jboss-cli accepts environment variables so you may have to substitute full patch for ${TEST_LOC} in jboss-cli)

Now run the client:

    cd $TEST_LOC/bundle/test-deadlock
    mvn exec:java


Then select “s” from the option list. 

If the bug is fixed, s will run, otherwise, there will be a deadlock and s will take a while to return, and will return with an error.

Comment 14 Jiří Bílek 2016-11-29 13:58:39 UTC
Thank you for reproducer.

A behavior of reproducer is the same for 6.4.10 as for 6.4.12. Insert into database is everytime succesfully finished but client is frozen.

Integration test is failing for both EAPs.

Reopened

Comment 16 Petr Penicka 2016-11-29 15:39:57 UTC
PM call: cloned into a new issue for 6.4.13 (or possibly 6.4.14 if Eng can't fix in time): https://bugzilla.redhat.com/show_bug.cgi?id=1399703

Closing this one as CLOSED NEXTRELEASE.