Created attachment 792093 [details] Attachement taken from the original BZ filed for EAP Description of problem: A WAR deployment containing its own infinispan (such as JDG helloworld quickstart) and also having persistence.xml with second level cache enabled fails to deploy on EAP 6.1 HA configuration (standalone-ha.xml or standalone-full-ha.xml) Version-Release number of selected component (if applicable): EAP 6.1.0, EAP 6.1.1.ER4. Note: This does not happen with EAP 6.0.1 so it's a regression. How reproducible: Always Steps to Reproduce: 1. Get JDG 6.1 helloworld-jdg quickstart (or see the attachment) 2. Add persistence.xml with 2lc, e.g.: <?xml version="1.0" encoding="UTF-8"?> <persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd" version="2.0"> <persistence-unit name="entityManager"> <jta-data-source>java:jboss/datasources/ExampleDS</jta-data-source> <properties> <property name="hibernate.hbm2ddl.auto" value="create-drop"/> <property name="hibernate.cache.use_second_level_cache" value="true"/> <property name="hibernate.cache.use_query_cache" value="true"/> </properties> </persistence-unit> </persistence> 3. deploy the helloworld-jdg on standalone-ha.xml or standalone-full-ha.xml configuration of EAP 6.1. This happens also when standalone.xml file is used to start EAP. Actual results: fails to deploy with: 10:26:20,759 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 53) MSC000001: Failed to start service jboss.persistenceunit."jboss-as-helloworld-jdg.war#entityManager": org.jboss.msc.service.StartException in service jboss.persistenceunit."jboss-as-helloworld-jdg.war#entityManager": javax.persistence.PersistenceException: [PersistenceUnit: entityManager] Unable to build EntityManagerFactory at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:103) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1] Caused by: javax.persistence.PersistenceException: [PersistenceUnit: entityManager] Unable to build EntityManagerFactory at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:930) at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:904) at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:92) at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.createContainerEntityManagerFactory(PersistenceUnitServiceImpl.java:200) at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.access$600(PersistenceUnitServiceImpl.java:57) at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:99) ... 4 more Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.spi.CacheImplementor] at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:186) at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:150) at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:131) at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:264) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1762) at org.hibernate.ejb.EntityManagerFactoryImpl.<init>(EntityManagerFactoryImpl.java:94) at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:920) ... 9 more Caused by: org.hibernate.cache.CacheException: Unable to start region factory at org.hibernate.cache.infinispan.InfinispanRegionFactory.start(InfinispanRegionFactory.java:323) at org.hibernate.internal.CacheImpl.<init>(CacheImpl.java:70) at org.hibernate.engine.spi.CacheInitiator.initiateService(CacheInitiator.java:40) at org.hibernate.engine.spi.CacheInitiator.initiateService(CacheInitiator.java:35) at org.hibernate.service.internal.SessionFactoryServiceRegistryImpl.initiateService(SessionFactoryServiceRegistryImpl.java:91) at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:176) ... 15 more Caused by: java.lang.ClassCastException: org.infinispan.remoting.ReplicationQueueImpl cannot be cast to org.infinispan.remoting.ReplicationQueue at org.infinispan.configuration.cache.LegacyConfigurationAdaptor.adapt(LegacyConfigurationAdaptor.java:306) at org.infinispan.manager.DefaultCacheManager.defineConfiguration(DefaultCacheManager.java:468) at org.infinispan.manager.DefaultCacheManager.defineConfiguration(DefaultCacheManager.java:443) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.defineConfiguration(DefaultEmbeddedCacheManager.java:77) at org.hibernate.cache.infinispan.InfinispanRegionFactory.defineGenericDataTypeCacheConfigurations(InfinispanRegionFactory.java:492) at org.hibernate.cache.infinispan.InfinispanRegionFactory.start(InfinispanRegionFactory.java:318) ... 20 more 10:26:20,968 ERROR [org.jboss.as.server] (management-handler-thread - 4) JBAS015870: Deploy of deployment "jboss-as-helloworld-jdg.war" was rolled back with the following failure message: {"JBAS014671: Failed services" => {"jboss.persistenceunit.\"jboss-as-helloworld-jdg.war#entityManager\"" => "org.jboss.msc.service.StartException in service jboss.persistenceunit.\"jboss-as-helloworld-jdg.war#entityManager\": javax.persistence.PersistenceException: [PersistenceUnit: entityManager] Unable to build EntityManagerFactory Caused by: javax.persistence.PersistenceException: [PersistenceUnit: entityManager] Unable to build EntityManagerFactory Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.spi.CacheImplementor] Caused by: org.hibernate.cache.CacheException: Unable to start region factory Caused by: java.lang.ClassCastException: org.infinispan.remoting.ReplicationQueueImpl cannot be cast to org.infinispan.remoting.ReplicationQueue"}} Expected results: Application deploys Additional info:
Mircea Markus <mmarkus> made a comment on jira ISPN-3496 ReplicationQueueImpl does implements ReplicationQueue, possibly a class loading issue. LegacyConfigurationAdaptor has been dropped in 6.0.0.Beta1, I'd be curious to see if this bug reproduces.
This bug is from LegacyConfigurationAdaptor#adapt(org.infinispan.config.Configuration) -- a new method added in Infinispan 5.2. The calls in this method to legacy.getClassLoader() return the wrong classloader. public ClassLoader getClassLoader() { if (cl != null) // The classloader has been set for this configuration return cl; else if (cl == null && globalConfiguration != null) // The classloader is not set for this configuration, and we have a global config return globalConfiguration.getClassLoader(); else // Return the default CL return Thread.currentThread().getContextClassLoader(); } The correct classloader to use was set by AS in the GlobalConfiguration. But at the point that DefaultCacheManager#defineCache calls LegacyConfigurationAdaptor#adapt, setGlobalConfiguration has not been called on the configuration object, so globalConfiguration is null, causing getClassLoader to incorrectly return the TCCL instead of the correct classloader. (In 5.1 setGlobalConfiguration was called later on during wireCache, but is no longer called at all in 5.2+)
This bug can be reproduced also with ISPN 6.0.0.Alpha4 deployed in EAP 6.1.1 (which uses 5.2.7.Final-redhat-1 internally). The exception is still the same.
Galder Zamarreño <galder.zamarreno> made a comment on jira ISPN-3496 Indeed, this is a classloading issue. The Infinispan classes provided by EAP and the Infinispan classes in the WAR deployment are getting messed up to lead to a CCE caused by different classloaders. This particular issue might be fixable, but there's a whole host of other issues that could surface. The question is: is this something we really want to be fixing? What is the problem with JDG Helloworld quickstart piggy backing on the provided Infinispan JAR by EAP? I'd expect that to work without problems.
Galder Zamarreño <galder.zamarreno> made a comment on jira ISPN-3496 @dereed, {quote} This bug is from LegacyConfigurationAdaptor#adapt(org.infinispan.config.Configuration) – a new method added in Infinispan 5.2. {quote} ^ Not a new method at all, it was there already in Infinispan [5.1.x|https://github.com/infinispan/infinispan/blob/5.1.8.Final/core/src/main/java/org/infinispan/configuration/cache/LegacyConfigurationAdaptor.java#L294]. I suspect the problem is due to different classloader() method defaults in 5.1.x vs 5.2.x. In 5.1.x, the default was [Thread.currentThread().getContextClassLoader();|https://github.com/infinispan/infinispan/blob/5.1.8.Final/core/src/main/java/org/infinispan/configuration/cache/ConfigurationBuilder.java#L7], whereas in 5.2.x is [null|https://github.com/infinispan/infinispan/blob/5.2.7.Final/core/src/main/java/org/infinispan/configuration/cache/ConfigurationBuilder.java#L34]. If null, Util class' classloader will be used.
Galder Zamarreño <galder.zamarreno> made a comment on jira ISPN-3496 @dereed, {quote} This bug is from LegacyConfigurationAdaptor#adapt(org.infinispan.config.Configuration) – a new method added in Infinispan 5.2. {quote} ^ Not a new method at all, it was there already in Infinispan [UPDATED] The default of ConfigurationBuilder has nothing to do here.
Galder Zamarreño <galder.zamarreno> made a comment on jira ISPN-3496 @dereed, {quote} This bug is from LegacyConfigurationAdaptor#adapt(org.infinispan.config.Configuration) – a new method added in Infinispan 5.2. {quote} ^ Not a new method at all, it was there already in Infinispan 5.1.x [UPDATED] The default of ConfigurationBuilder has nothing to do here.
> ^ Not a new method at all, it was there already in Infinispan 5.1.x You're right. What's new is that it's now called from DefaultCacheManager.defineConfiguration where setGlobalConfiguration has not been called on the configuration passed in. In order for LegacyConfigurationAdaptor#adapt to work correctly, DefaultCacheManager.defineConfiguration must call setGlobalConfiguration on the configuration before passing it to adapt (or something equivalent, to get the classloader set correctly).
> The question is: is this something we really want to be fixing? Definitely. We have to fix it, or you can't use JDG (or community Infinispan) in any deployment that also uses a hibernate 2nd level cache (or possibly other use cases). > What is the problem with JDG Helloworld quickstart piggy backing on the > provided Infinispan JAR by EAP? I'd expect that to work without problems. Then it's using the EAP Infinispan instead of JDG. (Which we don't support)
we can't rely on bundled ISPN in EAP/AS as that is private module and as such is not supported to use in applications, so there is no official supported way how to use ISPN in application with JPA layer, because of hibernate 2LC classloader conflict.
Mircea Markus <mmarkus> updated the status of jira ISPN-3496 to Resolved
Mircea Markus <mmarkus> updated the status of jira ISPN-3496 to Reopened
Pedro Ruivo <pedroruivo2> made a comment on jira ISPN-3496 [~mgencur] can you give it a try with https://github.com/pruivo/infinispan/tree/ISPN-3496_5_2_x ? I'm setting the global configuration before invoking the adapt... Also, the adapter as dropped in 6.0.beta1. Can you also give it a try with that version?
Dennis Reed <dereed> made a comment on jira ISPN-3496 PR2107 should fix the specific use case, but it's still different behavior than 5.1.x. 5.1.x: if ( configuration classloader set ) use it else if ( global configuration classloader set ) use it else use TCCL 5.2.x before patching: if ( configuration.classloader set ) use it else use TCCL 5.2.x after PR2107: use global configuration classloader
Dennis Reed <dereed> made a comment on jira ISPN-3496 PR2107 should fix the specific use case, but it's still different behavior than 5.1.x. 5.1.x: if ( configuration classloader set ) use it else if ( global configuration classloader set ) use it else use TCCL 5.2.x before patching: if ( configuration.classloader set ) use it else use TCCL 5.2.x after PR2107 (order change if set in multiple places): if ( global configuration classloader set ) use it else if ( configuration.classloader set ) use it else use TCCL
The fix for this bug was actually applied to 5.2.x ISPN branch so it does not affect JDG 6.2. There's no need for a fix in JDG 6.2