Bug 1202911

This change likely occurred because JBoss Modules (as of EAP 6.4, I believe) has switched to child-first loading.

This means that if the deployment is bundling a JAR which is also present as a container-provided module, the deployment JAR will take precedence as of this release.

Given that the deployment worked fine in 6.3, I would say that the problem is that the deployment should not have had the JAR in the first place; in 6.3 (unless something else changed), the deployment's JAR would not have been used, so what you're seeing isn't the container *starting* to use the container JAR instead of the user's; it's actually the container *stopping* using the container JAR in favor of the deployment one.

Hi David, thanks for reply.

The problem is that the JAR that causes the confict (hibernate-infinispan-4.2.17.Final-redhat-1.jar) is NOT present in the deployment WAR, so if I understood you comment correctly, this cannot be the case. Here is the content of the WAR (jar -tf):

WEB-INF/lib/
WEB-INF/beans.xml
WEB-INF/lib/arquillian-protocol.jar
WEB-INF/lib/mockito-all.jar
WEB-INF/lib/infinispan-cachestore-jpa.jar
WEB-INF/classes/com/jboss/datagrid/test/
WEB-INF/lib/infinispan-core-tests.jar
WEB-INF/classes/
WEB-INF/classes/config/
WEB-INF/classes/com/jboss/datagrid/test/jpastore/
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaConfigurationWrapperIT.class
WEB-INF/lib/infinispan-commons.jar
WEB-INF/lib/infinispan-cachestore-jpa-tests.jar
WEB-INF/lib/jboss-marshalling-river.jar
WEB-INF/lib/arquillian-testenricher-osgi.jar
WEB-INF/classes/com/jboss/datagrid/
WEB-INF/
WEB-INF/classes/com/jboss/datagrid/test/jpastore/utils/DeploymentBuilder.class
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreFunctionalWrapperIT.class
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreWrapperIT.class
WEB-INF/lib/mysql-connector-java-5.1.33-bin.jar
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreUserEntityWrapperIT.class
WEB-INF/classes/config/jpa-config-60.xml
WEB-INF/classes/com/jboss/datagrid/test/jpastore/utils/
WEB-INF/lib/arquillian-testenricher-msc.jar
WEB-INF/lib/hornetq-commons.jar
WEB-INF/lib/infinispan-core.jar
WEB-INF/lib/jboss-marshalling.jar
WEB-INF/lib/arquillian-testenricher-ejb.jar
META-INF/jboss-deployment-structure.xml
WEB-INF/classes/com/jboss/
WEB-INF/lib/jgroups.jar
WEB-INF/classes/com/
WEB-INF/lib/jta.jar
WEB-INF/lib/arquillian-testenricher-resource.jar
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreVehicleEntityWrapperIT.class
WEB-INF/lib/arquillian-testng.jar
META-INF/
WEB-INF/lib/arquillian-testenricher-cdi.jar
WEB-INF/lib/arquillian-core.jar
WEB-INF/classes/com/jboss/datagrid/test/jpastore/AbstractJpaITWrapper.class
WEB-INF/lib/junit.jar
WEB-INF/lib/arquillian-testenricher-initialcontext.jar

The hibernate-infinispan-4.2.17.Final-redhat-1.jar JAR doesn't *cause* the conflict, it merely *exposes* it.  The conflict is (most likely) that there is more than one org.infinispan.commands.module.ModuleCommandExtensions interface .class file visible to the deployment, causing the org.hibernate.cache.infinispan.util.CacheCommandExtensions class to link against the wrong one, making it not castable to the correct ModuleCommandExtensions type.  Since the app server would only ship at most one, the other (at least) one must be in the deployment somewhere.

From Jiri's deployment above, it is clear that there's ModuleCommandExtensions interface available in infinispan-core jar file that is bundled in the application.
And another one is made available to the application through org.hibernate -> org.infinispan dependency.

I've tried a different approach using JDG modules for EAP where infinispan-core is provided as a module, and removed all Infinispan dependencies from the war file.
I'm getting the same error unless I change module.xml from org.hibernate module in the following way:
<module name="org.infinispan" optional="true"/>  -> 
<module name="org.infinispan" optional="true" slot="jdg-6.5"/>

(JDG modules package JDG libraries under jdg-6.5 slot currently)

This can be a workaround.

I've witnesses several times that the AS injects Hibernate's dependency of the Infinispan module into the user deployment classpath; this has been problematic for extensions like Hibernate Search and while I don't know all the reasons behind this choice, I tried questioning them in the past but at least I can now provide some details on the issues.
I believe this is a long standing mistake of the structure of the Hibernate ORM modules which are exposing too much, combined with how both Hibernate ORM'code does classloadeing and how Infinispan does classloading.

An example of such findings is here:
http://lists.jboss.org/pipermail/hibernate-dev/2014-May/011437.html

But there are many more email discussions on the subject.

Scott Marlow mentioned he can't simply remove the dependency from the user classpath as that would trigger Infinispan classloader issues.
I'm not sure of which classloader issues, but I'm guessing it's the kind of:
http://lists.jboss.org/pipermail/infinispan-dev/2014-October/015549.html

For example, Infinispan wouldn't be able to load a custom configuration file from the app deployment (but that should be ok since you're supposed to configure that elsewhere?).

In short, I agree with David that this is probably just revealing a long standing issue.
To summarize the issue(s) at hand, I think the problem is that both Hibernate ORM and Infinispan cast a too wide net to load resources and extensions.

My previous link details how Infinispan is essentially attempting to load from any possible place it could think of, but Hibernate ORM does a similar thing:
https://github.com/hibernate/hibernate-orm/blob/4c690a8839113cd6850e1d2952e03a5030be1340/hibernate-core/src/main/java/org/hibernate/boot/registry/classloading/internal/ClassLoaderServiceImpl.java#L98

An additional problem is that, while both codebases try to load from all possible ClassLoaders they can get references to, they should have options for explicitly controlling the sources to be inspected.

With Hibernate ORM the ClassLoaderService can be overriden, so that would probably be nice to do from EAP's code (and this could improve performance as well by preventing it to attempt loading from locations which don't apply).
With Infinispan core, I suspect this is still not possible: Search still applies classloader workarounds.

The simple solution could be to stop exposing the EAP version of Infinispan to the user deployment classloader, even if it's using JPA. Hopefully that is no longer needed and this was done already? 

These emails I had sent are quite old, so bear with me if some of these details are outdated; I

What happens if you remove the deployment jars that are also defined as application server static modules?

As I wrote in comment 4, when the jars are NOT in the deployment but they are defined as static modules, the behavior is same unless I point org.hibernate module to the new static modules. When I don't do that, Hibernate still pulls the internal Infinispan libraries of EAP.

I just looked at the test archive. I found it surprising that the application is not depending on the JDG modules, but is including copies of the JDG jars in the deployment.
Is that supposed to be supported?

Yes:) Both ways are supported.

Thanks David, you nailed it, although I'm not sure how come it would work in the past (seemed a pretty wrong setup to me as well).

Let me clarify what's happening:

A JPA application will get the hibernate:main slot on its own classpath. That's automatically handled by the deployer.

This module includes hibernate-infinispan, which includes extension points to infinispan-core.
That extension point's class needs to be initialized with the infinispan:main version of the application server, as that's the only one which is compatible. This should work fine, as the hibernate:main module doesn't depend on the user deployment, so the extension point is linked to EAP's Infinispan.

When the Infinispan CacheManager is started using the bundled Infinispan version included in the user deployment, it has visibility to the hibernate module, finds the extension and will attempt to register it, but it's not compatible since it's being wired to the other Infinispan version.
That's exactly the stacktrace we see.

Scott reminded me of the property jboss.as.jpa.providerModule=application, that will prevent the hibernate:main module to be exposed to the deployment.
If you set that, it should then be possible to explicitly depend on org.hibernate:main but excluding org.hibernate.cache.infinispan, or simply exclude services so that your starting Infinispan won't find it.

Generally speaking, I strongly wish that hibernate-infinispan was separated from the main hibernate module as that would prevent such issues: opening a JIRA for WildFly.

I've opened this for the longer term solution:
https://issues.jboss.org/browse/WFLY-4517

In my previous comment I only pointed out to a possible workaround (which needs to be verified), as WFLY-4517 will require changes in the modules shipped by the application server.

Sanne, I will try the workaround but if I understand it correctly, when I use JDG modules for EAP, the property won't help as the libraries will be packaged as modules as well. Is that true?

Hey,
I can confirm that the suggested workaround works for me:

1) Add <property name="jboss.as.jpa.providerModule" value="application" /> into persistence.xml under <persistence-unit> / <properties>

2) Add the following jboss-deployment-structure.xml to your deployment:

<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.infinispan" />
        </exclusions>
        <dependencies>
            <module name="org.hibernate" />
        </dependencies>
    </deployment>
</jboss-deployment-structure>

3) When JDG modules for EAP are used, add to jboss-deployment-structure also the other JDG dependencies.

The workaround was tested locally so far and it seems to work. I'll configure the Jenkins job as well and confirm it there.

I can confirm that the workaround is working even in our Jenkins job. Thanks!

This is a regression caused by https://issues.jboss.org/browse/EAP6-225

The implicit dependency on org.hibernate for a JPA deployment was changed to include META-INF/services, where it did not before.

But hibernate-infinispan-*.jar's META-INF/services/org.infinispan.commands.module.ModuleCommandExtensions should not be exported, and is the cause of this issue.

To fix it, the change needs to be rolled back (which will also affect a couple of other META-INF/services in hibernate), or the org.hibernate module needs to be changed so it does not export that services file (WFLY-4517).

Slightly simpler workaround:

<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.hibernate" />
        </exclusions>
        <dependencies>
            <module name="org.hibernate" />
        </dependencies>
    </deployment>
</jboss-deployment-structure>

missed c04, lets get it into cp05

Yes I agree, WFLY-4517 should be the way to go; the solution was merged in WildFly but I don't think it was back-portable to EAP as essentially it requires to change some module names and the content of the app server configuration file: we updated the default configuration file but someone having an existing configuration file would need to make sure to fix his configuration file too.

Would requiring to change the contents of the configuration file be acceptable?

While it might look like it doesn't need to recompile any java code, it is possible that it relies on previous classloading fixes withing Hibernate ORM and/or Infinispan; that would need to be tested to identify what else needs backporting.

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

This depends on (as mentioned above)
https://bugzilla.redhat.com/show_bug.cgi?id=1290960 and there is a new PR that fixes this issue (more BZ cases were open for the same issue) -
https://github.com/jbossas/jboss-eap/pull/2764

Verified with EAP 6.4.9.CP.CR2

Retroactively bulk-closing issues from released EAP 6.4 cummulative patches.