Bug 1202911 - [GSS](6.4.z) Class loader loads extra services from modules
Summary: [GSS](6.4.z) Class loader loads extra services from modules
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Class Loading
Version: 6.4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: CR1
: EAP 6.4.9
Assignee: Dmitrii Tikhomirov
QA Contact: Peter Mackay
URL:
Whiteboard:
Depends On: 1290960
Blocks: eap649-payload
TreeView+ depends on / blocked
 
Reported: 2015-03-17 16:52 UTC by Jiri Holusa
Modified: 2019-11-14 06:39 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-17 12:56:29 UTC
Type: Bug


Attachments (Terms of Use)
Test that is being deployed on EAP. (12.22 MB, application/zip)
2015-03-17 16:52 UTC, Jiri Holusa
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2218051 None None None 2016-07-06 13:32:39 UTC

Description Jiri Holusa 2015-03-17 16:52:41 UTC
Created attachment 1002878 [details]
Test that is being deployed on EAP.

We started testing JDG JPA cache store with EAP 6.4.Beta and came up with this behaviour. 

The Jenkins job with console output with the exception can be found here: 
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/jdg-func-db-compatibility/DATABASE=postgresql92,TEST=jpa,jdk=java17_default,label_exp=%28RHEL5||RHEL6%29&&x86_64&&mem4G/28/console

This test is fully functional and passes with EAP 6.3, but with EAP 6.4.Beta following exception is thrown:
Caused by: java.util.ServiceConfigurationError: org.infinispan.commands.module.ModuleCommandExtensions: Provider org.hibernate.cache.infinispan.util.CacheCommandExtensions not a subtype 
(see the link for the full stacktrace)

I debugged the issue and the problem seems to be that EAP loads hibernate-infinispan-4.2.17.Final-redhat-1.jar and parses its /META-INF/services, causing loading of org.hibernate.cache.infinispan.util.CacheCommandExtensions, which causes the problem.

However, we don't see a reason, why EAP loads this module jar in a first place. All libraries are bundled in the WAR, so this jar is not required. Note that the same test is fine with EAP 6.3. When I try to delete the services folder from hibernate-infinispan-4.2.17.Final-redhat-1.jar, the CacheCommandExtensions is no longer loaded and everything is fine. By the way, when debugging EAP 6.3, it doesn't load this service at all, it just doesn't see it. 

Both those module jars in 6.3 and 6.4.Beta are identical. I attached the deployment WAR, which is an Arqullian test.

Comment 1 David M. Lloyd 2015-03-17 16:58:16 UTC
This change likely occurred because JBoss Modules (as of EAP 6.4, I believe) has switched to child-first loading.

This means that if the deployment is bundling a JAR which is also present as a container-provided module, the deployment JAR will take precedence as of this release.

Given that the deployment worked fine in 6.3, I would say that the problem is that the deployment should not have had the JAR in the first place; in 6.3 (unless something else changed), the deployment's JAR would not have been used, so what you're seeing isn't the container *starting* to use the container JAR instead of the user's; it's actually the container *stopping* using the container JAR in favor of the deployment one.

Comment 2 Jiri Holusa 2015-04-07 13:46:06 UTC
Hi David, thanks for reply.

The problem is that the JAR that causes the confict (hibernate-infinispan-4.2.17.Final-redhat-1.jar) is NOT present in the deployment WAR, so if I understood you comment correctly, this cannot be the case. Here is the content of the WAR (jar -tf):

WEB-INF/lib/
WEB-INF/beans.xml
WEB-INF/lib/arquillian-protocol.jar
WEB-INF/lib/mockito-all.jar
WEB-INF/lib/infinispan-cachestore-jpa.jar
WEB-INF/classes/com/jboss/datagrid/test/
WEB-INF/lib/infinispan-core-tests.jar
WEB-INF/classes/
WEB-INF/classes/config/
WEB-INF/classes/com/jboss/datagrid/test/jpastore/
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaConfigurationWrapperIT.class
WEB-INF/lib/infinispan-commons.jar
WEB-INF/lib/infinispan-cachestore-jpa-tests.jar
WEB-INF/lib/jboss-marshalling-river.jar
WEB-INF/lib/arquillian-testenricher-osgi.jar
WEB-INF/classes/com/jboss/datagrid/
WEB-INF/
WEB-INF/classes/com/jboss/datagrid/test/jpastore/utils/DeploymentBuilder.class
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreFunctionalWrapperIT.class
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreWrapperIT.class
WEB-INF/lib/mysql-connector-java-5.1.33-bin.jar
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreUserEntityWrapperIT.class
WEB-INF/classes/config/jpa-config-60.xml
WEB-INF/classes/com/jboss/datagrid/test/jpastore/utils/
WEB-INF/lib/arquillian-testenricher-msc.jar
WEB-INF/lib/hornetq-commons.jar
WEB-INF/lib/infinispan-core.jar
WEB-INF/lib/jboss-marshalling.jar
WEB-INF/lib/arquillian-testenricher-ejb.jar
META-INF/jboss-deployment-structure.xml
WEB-INF/classes/com/jboss/
WEB-INF/lib/jgroups.jar
WEB-INF/classes/com/
WEB-INF/lib/jta.jar
WEB-INF/lib/arquillian-testenricher-resource.jar
WEB-INF/classes/com/jboss/datagrid/test/jpastore/JpaStoreVehicleEntityWrapperIT.class
WEB-INF/lib/arquillian-testng.jar
META-INF/
WEB-INF/lib/arquillian-testenricher-cdi.jar
WEB-INF/lib/arquillian-core.jar
WEB-INF/classes/com/jboss/datagrid/test/jpastore/AbstractJpaITWrapper.class
WEB-INF/lib/junit.jar
WEB-INF/lib/arquillian-testenricher-initialcontext.jar

Comment 3 David M. Lloyd 2015-04-07 13:53:49 UTC
The hibernate-infinispan-4.2.17.Final-redhat-1.jar JAR doesn't *cause* the conflict, it merely *exposes* it.  The conflict is (most likely) that there is more than one org.infinispan.commands.module.ModuleCommandExtensions interface .class file visible to the deployment, causing the org.hibernate.cache.infinispan.util.CacheCommandExtensions class to link against the wrong one, making it not castable to the correct ModuleCommandExtensions type.  Since the app server would only ship at most one, the other (at least) one must be in the deployment somewhere.

Comment 4 Martin Gencur 2015-04-14 08:29:14 UTC
From Jiri's deployment above, it is clear that there's ModuleCommandExtensions interface available in infinispan-core jar file that is bundled in the application.
And another one is made available to the application through org.hibernate -> org.infinispan dependency.

I've tried a different approach using JDG modules for EAP where infinispan-core is provided as a module, and removed all Infinispan dependencies from the war file.
I'm getting the same error unless I change module.xml from org.hibernate module in the following way:
<module name="org.infinispan" optional="true"/>  -> 
<module name="org.infinispan" optional="true" slot="jdg-6.5"/>

(JDG modules package JDG libraries under jdg-6.5 slot currently)

This can be a workaround.

Comment 5 Sanne Grinovero 2015-04-14 09:27:24 UTC
I've witnesses several times that the AS injects Hibernate's dependency of the Infinispan module into the user deployment classpath; this has been problematic for extensions like Hibernate Search and while I don't know all the reasons behind this choice, I tried questioning them in the past but at least I can now provide some details on the issues.
I believe this is a long standing mistake of the structure of the Hibernate ORM modules which are exposing too much, combined with how both Hibernate ORM'code does classloadeing and how Infinispan does classloading.

An example of such findings is here:
http://lists.jboss.org/pipermail/hibernate-dev/2014-May/011437.html

But there are many more email discussions on the subject.

Scott Marlow mentioned he can't simply remove the dependency from the user classpath as that would trigger Infinispan classloader issues.
I'm not sure of which classloader issues, but I'm guessing it's the kind of:
http://lists.jboss.org/pipermail/infinispan-dev/2014-October/015549.html

For example, Infinispan wouldn't be able to load a custom configuration file from the app deployment (but that should be ok since you're supposed to configure that elsewhere?).

In short, I agree with David that this is probably just revealing a long standing issue.
To summarize the issue(s) at hand, I think the problem is that both Hibernate ORM and Infinispan cast a too wide net to load resources and extensions.

My previous link details how Infinispan is essentially attempting to load from any possible place it could think of, but Hibernate ORM does a similar thing:
https://github.com/hibernate/hibernate-orm/blob/4c690a8839113cd6850e1d2952e03a5030be1340/hibernate-core/src/main/java/org/hibernate/boot/registry/classloading/internal/ClassLoaderServiceImpl.java#L98

An additional problem is that, while both codebases try to load from all possible ClassLoaders they can get references to, they should have options for explicitly controlling the sources to be inspected.

With Hibernate ORM the ClassLoaderService can be overriden, so that would probably be nice to do from EAP's code (and this could improve performance as well by preventing it to attempt loading from locations which don't apply).
With Infinispan core, I suspect this is still not possible: Search still applies classloader workarounds.

The simple solution could be to stop exposing the EAP version of Infinispan to the user deployment classloader, even if it's using JPA. Hopefully that is no longer needed and this was done already? 

These emails I had sent are quite old, so bear with me if some of these details are outdated; I

Comment 6 Scott Marlow 2015-04-14 15:51:44 UTC
What happens if you remove the deployment jars that are also defined as application server static modules?

Comment 7 Martin Gencur 2015-04-14 15:55:42 UTC
As I wrote in comment 4, when the jars are NOT in the deployment but they are defined as static modules, the behavior is same unless I point org.hibernate module to the new static modules. When I don't do that, Hibernate still pulls the internal Infinispan libraries of EAP.

Comment 8 Sanne Grinovero 2015-04-14 16:07:26 UTC
I just looked at the test archive. I found it surprising that the application is not depending on the JDG modules, but is including copies of the JDG jars in the deployment.
Is that supposed to be supported?

Comment 9 Martin Gencur 2015-04-14 16:08:14 UTC
Yes:) Both ways are supported.

Comment 10 Sanne Grinovero 2015-04-14 16:52:10 UTC
Thanks David, you nailed it, although I'm not sure how come it would work in the past (seemed a pretty wrong setup to me as well).

Let me clarify what's happening:

A JPA application will get the hibernate:main slot on its own classpath. That's automatically handled by the deployer.

This module includes hibernate-infinispan, which includes extension points to infinispan-core.
That extension point's class needs to be initialized with the infinispan:main version of the application server, as that's the only one which is compatible. This should work fine, as the hibernate:main module doesn't depend on the user deployment, so the extension point is linked to EAP's Infinispan.

When the Infinispan CacheManager is started using the bundled Infinispan version included in the user deployment, it has visibility to the hibernate module, finds the extension and will attempt to register it, but it's not compatible since it's being wired to the other Infinispan version.
That's exactly the stacktrace we see.

Scott reminded me of the property jboss.as.jpa.providerModule=application, that will prevent the hibernate:main module to be exposed to the deployment.
If you set that, it should then be possible to explicitly depend on org.hibernate:main but excluding org.hibernate.cache.infinispan, or simply exclude services so that your starting Infinispan won't find it.

Generally speaking, I strongly wish that hibernate-infinispan was separated from the main hibernate module as that would prevent such issues: opening a JIRA for WildFly.

Comment 11 Sanne Grinovero 2015-04-14 17:04:13 UTC
I've opened this for the longer term solution:
https://issues.jboss.org/browse/WFLY-4517

In my previous comment I only pointed out to a possible workaround (which needs to be verified), as WFLY-4517 will require changes in the modules shipped by the application server.

Comment 12 Martin Gencur 2015-04-15 07:15:49 UTC
Sanne, I will try the workaround but if I understand it correctly, when I use JDG modules for EAP, the property won't help as the libraries will be packaged as modules as well. Is that true?

Comment 13 Martin Gencur 2015-04-15 07:55:08 UTC
Hey,
I can confirm that the suggested workaround works for me:

1) Add <property name="jboss.as.jpa.providerModule" value="application" /> into persistence.xml under <persistence-unit> / <properties>

2) Add the following jboss-deployment-structure.xml to your deployment:

<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.infinispan" />
        </exclusions>
        <dependencies>
            <module name="org.hibernate" />
        </dependencies>
    </deployment>
</jboss-deployment-structure>

3) When JDG modules for EAP are used, add to jboss-deployment-structure also the other JDG dependencies.

Comment 14 Jiri Holusa 2015-04-16 08:38:19 UTC
The workaround was tested locally so far and it seems to work. I'll configure the Jenkins job as well and confirm it there.

Comment 15 Jiri Holusa 2015-04-16 12:15:58 UTC
I can confirm that the workaround is working even in our Jenkins job. Thanks!

Comment 16 dereed 2015-06-23 16:58:38 UTC
This is a regression caused by https://issues.jboss.org/browse/EAP6-225

The implicit dependency on org.hibernate for a JPA deployment was changed to include META-INF/services, where it did not before.

But hibernate-infinispan-*.jar's META-INF/services/org.infinispan.commands.module.ModuleCommandExtensions should not be exported, and is the cause of this issue.

To fix it, the change needs to be rolled back (which will also affect a couple of other META-INF/services in hibernate), or the org.hibernate module needs to be changed so it does not export that services file (WFLY-4517).

Slightly simpler workaround:

<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.hibernate" />
        </exclusions>
        <dependencies>
            <module name="org.hibernate" />
        </dependencies>
    </deployment>
</jboss-deployment-structure>

Comment 17 Shay Matasaro 2015-10-16 14:12:15 UTC
missed c04, lets get it into cp05

Comment 20 Sanne Grinovero 2015-10-16 15:02:26 UTC
Yes I agree, WFLY-4517 should be the way to go; the solution was merged in WildFly but I don't think it was back-portable to EAP as essentially it requires to change some module names and the content of the app server configuration file: we updated the default configuration file but someone having an existing configuration file would need to make sure to fix his configuration file too.

Would requiring to change the contents of the configuration file be acceptable?

While it might look like it doesn't need to recompile any java code, it is possible that it relies on previous classloading fixes withing Hibernate ORM and/or Infinispan; that would need to be tested to identify what else needs backporting.

Comment 31 Mike McCune 2016-03-28 22:31:02 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions

Comment 35 Miroslav Sochurek 2016-06-09 11:54:13 UTC
This depends on (as mentioned above)
https://bugzilla.redhat.com/show_bug.cgi?id=1290960 and there is a new PR that fixes this issue (more BZ cases were open for the same issue) -
https://github.com/jbossas/jboss-eap/pull/2764

Comment 36 Peter Mackay 2016-06-30 09:49:55 UTC
Verified with EAP 6.4.9.CP.CR2

Comment 38 Petr Penicka 2017-01-17 12:56:29 UTC
Retroactively bulk-closing issues from released EAP 6.4 cummulative patches.


Note You need to log in before you can comment on or make changes to this bug.