In case of transactional invocations spanning both EAP6 and EAP7 nodes, EAP7 server may send beforeCompletion message to the master transaction - this should be ignored.

The issue seems to be caused by change in the way in which the old and the new client behaviour regarding various protocol messages, specifically those dealing with imported transaction. The simple scenario with which I was able to reproduce the errors from the scenarios above are...
node1 -> node2 -> node1
... in both EAP7->EAP6->EAP7 and EAP6->EAP7->EAP6 combinations.

If both nodes are working on the same EAP version it is as follows:

EAP6.4:
1. node 1 (starts top transaction A)
2. node 2 (starts subordinate transaction B)
3. node 1 (continues operation in transaction A)
4. node 1 sends commit (no beforeCompletion call) to node 2 (A->B)
5. node 2 sends commit (no beforeCompletion call) to node 1 (B->A -> this commit is ignored (https://github.com/jbossas/jboss-eap/blob/6.4.x/ejb3/src/main/java/org/jboss/as/ejb3/remote/protocol/versionone/XidTransactionCommitTask.java#L68)

EAP7.2:
1. node 1 (starts top transaction A)
2. node 2 (starts subordinate transaction B (node2 learns that there is a master transaction on node1 (https://github.com/wildfly/wildfly-transaction-client/blob/master/src/main/java/org/wildfly/transaction/client/OutflowHandleManager.java#L130))
3. node 1 (transaction continues in A)
4. node 1 sends beforeCompletion and commit to node 2 (A->B)
5. B committed (node2 does not send beforeCompletion nor commit to node 1)
6. A committed

EAP7.2 -> EAP6.4 -> EAP7.2
1. node 1 (starts top transaction A)
2. node 2 (starts subordinate transaction B (node1 doesn't know that there is master on node2)
3. node 1 (transaction continues in A)
4. node 1 sends commit to node 2 (A->B)
5. node 2 sends commit to node 1 (B->A) (FAILURE - NOT SUBORDINATE (https://github.com/wildfly/wildfly-transaction-client/blob/master/src/main/java/org/wildfly/transaction/client/provider/jboss/JBossLocalTransactionProvider.java#L495))

This scenario can be worked around by logging warning message instead of throwing an error (https://github.com/wildfly/wildfly-transaction-client/blob/master/src/main/java/org/wildfly/transaction/client/provider/jboss/JBossLocalTransactionProvider.java#L495) - the transactions are finished correctly. I have create the PR to deal with it: https://github.com/wildfly/wildfly-transaction-client/pull/105.

EAP6.4 -> EAP7.2 -> EAP6.4
1. node 1 (starts top transaction A)
2. node 2 (starts subordinate B (node 2 can't learn that this is not subordinate in this version of the protocol))
3. node 1 (continues operation in A)
3. node 1 sends commit to node 2 (A->B)
4. node 2 sends beforeCompletion to node 1 (FAILURE NOT SUBORDINATE (https://github.com/jbossas/jboss-eap/blob/6.4.x/ejb3/src/main/java/org/jboss/as/ejb3/remote/protocol/versionone/XidTransactionBeforeCompletionTask.java#L46))

This scenario can again be worked around by warning message instead of throwing an error (in the same was as in XidTransactionCommitTask). I have created a PR to deal with it: https://github.com/jbossas/jboss-eap/pull/3145

After those PRs are applied all scenarios from the reproducer seem to work correctly. Michael Musgrove Ondrej Chaloupka could you please review whether this analysis is correct and whether you consider those above fixes as correct as well?

As a side note, there are some marshalling ClassNotFoundExceptions (f.e. EjbIdentifier which indeed is not present in EAP6.4 yet). Those disseapear if the errors above are eliminated so this is not a main problem but they may need some investigation at some point.