Bug 1069606 (JON3-9)

Summary: Rebase JON onto EAP 6.3
Product: [JBoss] JBoss Operations Network Reporter: Heiko W. Rupp <hrupp>
Component: Core ServerAssignee: John Mazzitelli <mazz>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: JON 3.3.0CC: asantos, jcacek, kkhan, lkrejci, mazz
Target Milestone: DR01   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-11 14:02:32 UTC Type: Task
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1081675, 1082072, 1082467, 1098577, 1100361, 1102332    
Bug Blocks:    
Attachments:
Description Flags
test war with service activator showing class-not-found error
none
policy file to use via -Djava.security.policy== none

Description Heiko W. Rupp 2014-02-25 11:35:33 UTC 
We need to use EAP 6.3 as base for JON / RHQ.
Same scenario as for 6.1 applies
Comment 1 John Mazzitelli 2014-03-24 17:22:30 UTC 
First, have to update gnu-getopt dependency because the 1.0.13 dep that everyone knows and loves has a bad MANIFEST.MF and it can't be loaded by EAP (deployment fails). See the manifest in this jar: https://repository.jboss.org/nexus/content/groups/public/gnu-getopt/getopt/1.0.13/getopt-1.0.13.jar -- so we have to move to gnu.getopt version java-getopt-1.0.13.redhat-4 (which ships with current build of 6.3).

But now when I run, I get a very odd issue - exception when trying to log, of all things:

Caused by: java.util.NoSuchElementException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:897) [rt.jar:1.7.0_11]
        at java.util.HashMap$KeyIterator.next(HashMap.java:928) [rt.jar:1.7.0_11]
        at org.jboss.as.jmx.PluggableMBeanServerImpl$LogAction.getCallerUserId(PluggableMBeanServerImpl.java:1249)
        at org.jboss.as.jmx.PluggableMBeanServerImpl$LogAction.doLog(PluggableMBeanServerImpl.java:1232)
        at org.jboss.as.jmx.PluggableMBeanServerImpl$LogAction.run(PluggableMBeanServerImpl.java:1224)
        at org.jboss.as.jmx.PluggableMBeanServerImpl$LogAction.run(PluggableMBeanServerImpl.java:1202)
        at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_11]
        at org.jboss.as.jmx.PluggableMBeanServerImpl.log(PluggableMBeanServerImpl.java:1155)
        at org.jboss.as.jmx.MBeanServerAuditLogRecordFormatter.log(MBeanServerAuditLogRecordFormatter.java:330)
        at org.jboss.as.jmx.MBeanServerAuditLogRecordFormatter.getMBeanInfo(MBeanServerAuditLogRecordFormatter.java:259)
        at org.jboss.as.jmx.PluggableMBeanServerImpl.getMBeanInfo(PluggableMBeanServerImpl.java:593)
        at org.jboss.as.jmx.PluggableMBeanServerImpl.getMBeanInfo(PluggableMBeanServerImpl.java:569)
        at org.jboss.mx.util.MBeanProxyExt.init(MBeanProxyExt.java:407) [jboss-jmx-4.2.3.GA.jar:4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807192035)]

Still trying to figure it out.
Comment 2 John Mazzitelli 2014-03-24 17:26:36 UTC 
Looks like we're stuck until this is fixed:

https://issues.jboss.org/browse/WFLY-3124
Comment 3 John Mazzitelli 2014-03-24 18:04:56 UTC 
to develop this, I created a github branch: feature/bz1069606-rebase-eap-63
Comment 4 John Mazzitelli 2014-03-26 19:02:08 UTC 
using the latest build, and now I'm not getting as far as I did before! Crapping out during installation - looks like more garbage to do with the logging subsystem:

14:58:33,035 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 29) MSC000001: Failed to start service jboss.persistenceunit."rhq.ear/rhq-core-domain-ejb3.jar#rhqpu": org.jboss.msc.service.StartException in service jboss.persistenceunit."rhq.ear/rhq-core-domain-ejb3.jar#rhqpu": javax.persistence.PersistenceException: [PersistenceUnit: rhqpu] Unable to build EntityManagerFactory
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:103)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: rhqpu] Unable to build EntityManagerFactory
	at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:924)
	at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:899)
	at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:76)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.createContainerEntityManagerFactory(PersistenceUnitServiceImpl.java:200)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.access$600(PersistenceUnitServiceImpl.java:57)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:99)
	... 4 more
Caused by: org.hibernate.MappingException: Could not instantiate persister org.hibernate.persister.entity.SingleTableEntityPersister
	at org.hibernate.persister.internal.PersisterFactoryImpl.create(PersisterFactoryImpl.java:174)
	at org.hibernate.persister.internal.PersisterFactoryImpl.createEntityPersister(PersisterFactoryImpl.java:135)
	at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:389)
	at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1797)
	at org.hibernate.ejb.EntityManagerFactoryImpl.<init>(EntityManagerFactoryImpl.java:96)
	at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:914)
	... 9 more
Caused by: java.lang.ExceptionInInitializerError
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_11]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_11]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_11]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525) [rt.jar:1.7.0_11]
	at org.hibernate.engine.internal.UnsavedValueFactory.instantiate(UnsavedValueFactory.java:46)
	at org.hibernate.engine.internal.UnsavedValueFactory.getUnsavedIdentifierValue(UnsavedValueFactory.java:68)
	at org.hibernate.tuple.PropertyFactory.buildIdentifierProperty(PropertyFactory.java:75)
	at org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:145)
	at org.hibernate.persister.entity.AbstractEntityPersister.<init>(AbstractEntityPersister.java:506)
	at org.hibernate.persister.entity.SingleTableEntityPersister.<init>(SingleTableEntityPersister.java:146)
	at sun.reflect.GeneratedConstructorAccessor69.newInstance(Unknown Source) [:1.7.0_11]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_11]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525) [rt.jar:1.7.0_11]
	at org.hibernate.persister.internal.PersisterFactoryImpl.create(PersisterFactoryImpl.java:163)
	... 14 more
Caused by: java.security.AccessControlException: access denied ("java.util.logging.LoggingPermission" "control")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:366) [rt.jar:1.7.0_11]
	at java.security.AccessController.checkPermission(AccessController.java:560) [rt.jar:1.7.0_11]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) [rt.jar:1.7.0_11]
	at org.jboss.logmanager.LogContext.checkSecurityAccess(LogContext.java:345) [jboss-logmanager-1.5.2.Final-redhat-1.jar:1.5.2.Final-redhat-1]
	at org.jboss.logmanager.Logger.attachIfAbsent(Logger.java:198) [jboss-logmanager-1.5.2.Final-redhat-1.jar:1.5.2.Final-redhat-1]
	at org.slf4j.impl.Slf4jLoggerFactory.getLogger(Slf4jLoggerFactory.java:44)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:278)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:274)
	at org.rhq.core.domain.configuration.ObfuscatedPropertySimple.<clinit>(ObfuscatedPropertySimple.java:58)
	... 28 more
Comment 5 John Mazzitelli 2014-03-26 19:15:15 UTC 
This is the problem - it looks like the EAP 6.3 (WildFly8) now has some kind of security manager that either is or should be installed/configured. Need to figure out what the secret sauce is to fix this:

Caused by: java.security.AccessControlException: access denied ("java.util.logging.LoggingPermission" "control")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:366) [rt.jar:1.7.0_11]
	at java.security.AccessController.checkPermission(AccessController.java:560) [rt.jar:1.7.0_11]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) [rt.jar:1.7.0_11]
	at org.jboss.logmanager.LogContext.checkSecurityAccess(LogContext.java:345) [jboss-logmanager-1.5.2.Final-redhat-1.jar:1.5.2.Final-redhat-1]
	at org.jboss.logmanager.Logger.attachIfAbsent(Logger.java:198) [jboss-logmanager-1.5.2.Final-redhat-1.jar:1.5.2.Final-redhat-1]
	at org.slf4j.impl.Slf4jLoggerFactory.getLogger(Slf4jLoggerFactory.java:44)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:278)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:274)
	at org.rhq.core.domain.configuration.ObfuscatedPropertySimple.<clinit>(ObfuscatedPropertySimple.java:58)
	... 28 more
Comment 6 John Mazzitelli 2014-03-26 20:20:30 UTC 
this appears to be my issue:

https://community.jboss.org/message/825205#825205

I don't know why I need to add this new security-manager and I don't know why the EAP build doesn't include this already in the config.
Comment 7 John Mazzitelli 2014-03-27 18:29:06 UTC 
(In reply to John Mazzitelli from comment #6)
> this appears to be my issue:
> 
> https://community.jboss.org/message/825205#825205
> 
> I don't know why I need to add this new security-manager and I don't know
> why the EAP build doesn't include this already in the config.

That thread deals with WildFly 8 - so the solution isn't applicable to EAP 6.3.

The dev team will fix this in the next build - from James Perkins:

   "Can you try replacing the org.jboss.slf4j:slf4j-jboss-logmanager with version 1.0.3.GA and see if the problem goes away? ... just download it from https://repository.jboss.org/nexus/content/groups/public/org/jboss/slf4j/slf4j-jboss-logmanager/1.0.3.GA/slf4j-jboss-logmanager-1.0.3.GA.jar and replace the slf4j-jboss-logmanager in $MODULES_PATH/org/slf4j/impl/main with the new version. Oh and edit the module.xml to point to it"

When I did that, I no longer get this security manager error. I got further, but still errors occurred during startup due to another issue. I'll log a new comment on that new error.
Comment 8 John Mazzitelli 2014-03-27 18:30:15 UTC 
Now getting this during startup (assuming slf4j is moved to 1.0.3.GA) - this appears to be Hibernate not liking a named query:

14:15:26,548 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 1) MSC000001: Failed to start service jboss.persistenceunit."rhq.ear/rhq-core-domain-ejb3.jar#rhqpu": org.jboss.msc.service.StartException in service jboss.persistenceunit."rhq.ear/rhq-core-domain-ejb3.jar#rhqpu": java.lang.NoClassDefFoundError: org/rhq/core/domain/resource/ResourceTypeFacet
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:103)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]
Caused by: java.lang.NoClassDefFoundError: org/rhq/core/domain/resource/ResourceTypeFacet
	at org.rhq.core.domain.resource.composite.ResourceFacets.initEnum(ResourceFacets.java:206)
	at org.rhq.core.domain.resource.composite.ResourceFacets.<init>(ResourceFacets.java:76)
	at org.rhq.core.domain.resource.composite.ResourceFacets.<clinit>(ResourceFacets.java:42)
	at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0_11]
	at java.lang.Class.forName(Class.java:264) [rt.jar:1.7.0_11]
	at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:148)
	at org.hibernate.internal.SessionFactoryImpl.getImportedClassName(SessionFactoryImpl.java:1323)
	at org.hibernate.hql.internal.ast.util.SessionFactoryHelper.getImportedClassName(SessionFactoryHelper.java:109)
	at org.hibernate.hql.internal.ast.tree.ConstructorNode.resolveConstructor(ConstructorNode.java:173)
	at org.hibernate.hql.internal.ast.tree.ConstructorNode.prepare(ConstructorNode.java:153)
	at org.hibernate.hql.internal.ast.HqlSqlWalker.processConstructor(HqlSqlWalker.java:1020)
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.selectExpr(HqlSqlBaseWalker.java:2315)
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.selectExprList(HqlSqlBaseWalker.java:2181)
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.selectClause(HqlSqlBaseWalker.java:1469)
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.query(HqlSqlBaseWalker.java:571)
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.selectStatement(HqlSqlBaseWalker.java:299)
	at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.statement(HqlSqlBaseWalker.java:247)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.analyze(QueryTranslatorImpl.java:250)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.doCompile(QueryTranslatorImpl.java:185)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.compile(QueryTranslatorImpl.java:138)
	at org.hibernate.engine.query.spi.HQLQueryPlan.<init>(HQLQueryPlan.java:104)
	at org.hibernate.engine.query.spi.HQLQueryPlan.<init>(HQLQueryPlan.java:79)
	at org.hibernate.engine.query.spi.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:168)
	at org.hibernate.internal.SessionFactoryImpl.checkNamedQueries(SessionFactoryImpl.java:1086)
	at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:523)
	at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1797)
	at org.hibernate.ejb.EntityManagerFactoryImpl.<init>(EntityManagerFactoryImpl.java:96)
	at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:914)
	at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:899)
	at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:76)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.createContainerEntityManagerFactory(PersistenceUnitServiceImpl.java:200)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.access$600(PersistenceUnitServiceImpl.java:57)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:99)
	... 4 more
Caused by: java.lang.ClassNotFoundException: org.rhq.core.domain.resource.ResourceTypeFacet from [Module "deployment.rhq.ear.rhq-core-domain-ejb3.jar:main" from Service Module Loader]
	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134) [jboss-modules.jar:1.3.3.Final-redhat-1]
	... 37 more
Comment 9 John Mazzitelli 2014-03-27 20:42:50 UTC 
(In reply to John Mazzitelli from comment #8)
> Caused by: java.lang.NoClassDefFoundError:
> org/rhq/core/domain/resource/ResourceTypeFacet
> 	at
> org.rhq.core.domain.resource.composite.ResourceFacets.
> initEnum(ResourceFacets.java:206)

To get around this, I refactored ResourceFacets so initEnum is not called from its constructor. If I change it to be called from the getter, this error no longer occurs. I may just commit that to get this to work. Its not a big deal.
Comment 10 John Mazzitelli 2014-03-27 20:44:47 UTC 
Created attachment 879656 [details]
test war with service activator showing class-not-found error

Next hurdle to jump, now that the hibernate problem is passed.

I didn't see this in the JON Server until I put DR6 under the hood. I have a simple service activator defined in the war. To try to isolate the problem, I created a stripped down test war that mimics what JON does and put it in JON Server's standalone/deployments directory. When I run, I get this exception. You can see that my service is activated (that's my service spitting out the "$$$$$ SERVICE ACTIVATING!" message). But when it tries adding a dependency via the service activator context, I get this class-not-found error. (the source code of this service is in the attached war - its a single .java file). I don't understand why it would not be able to find this class - its defined within the ServiceActivator class itself (its an inner class). Throw this test war in a stock D5 build, and it works fine, no exception.

16:14:50,494 INFO  [stdout] (MSC service thread 1-2) $$$$$$$$$$$$$$$$ SERVICE ACTIVATING!                                    
16:14:50,496 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.deployment.unit."test-service.war".INSTALL: org.jboss.msc.service.StartException in service jboss.deployment.unit."test-service.war".INSTALL: JBAS018733: Failed to process phase INSTALL of deployment "test-service.war"                                           
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:166) [jboss-as-server-7.4.0.Final-redhat-6.jar:7.4.0.Final-redhat-6]                                                                           
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final.jar:1.1.5.Final]                                                                                                    
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final.jar:1.1.5.Final]                                                                                                             
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]                 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]                 
        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]                                                           
Caused by: java.lang.NoClassDefFoundError: service/ManagementService$GetModelControllerService                               
        at service.ManagementService.activate(ManagementService.java:41)                                                     
        at org.jboss.as.server.deployment.service.ServiceActivatorProcessor.deploy(ServiceActivatorProcessor.java:69) [jboss-as-server-7.4.0.Final-redhat-6.jar:7.4.0.Final-redhat-6]                                                                     
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159) [jboss-as-server-7.4.0.Final-redhat-6.jar:7.4.0.Final-redhat-6]                                                                           
        ... 5 more                                                                                                           
Caused by: java.lang.ClassNotFoundException: service.ManagementService$GetModelControllerService from [Module "deployment.test-service.war:main" from Service Module Loader]                                                                              
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213) [jboss-modules.jar:1.3.3.Final-redhat-1]
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459) [jboss-modules.jar:1.3.3.Final-redhat-1]                                                                                                     
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408) [jboss-modules.jar:1.3.3.Final-redhat-1]                                                                                                       
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389) [jboss-modules.jar:1.3.3.Final-redhat-1]                                                                                                              
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134) [jboss-modules.jar:1.3.3.Final-redhat-1]                                                                                                                     
        ... 8 more
Comment 11 John Mazzitelli 2014-03-27 21:10:46 UTC 
from josef cacek:

   try to add following switch to your JAVA_OPTS: "-Djboss.modules.policy-permissions=true" If it helps, then the problem is related to: https://bugzilla.redhat.com/show_bug.cgi?id=1080826

That did help.
Comment 12 John Mazzitelli 2014-03-27 21:19:15 UTC 
the server still seems to work but during startup, I see this error that we should clean up:

17:16:37,961 ERROR [org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl] (Periodic Recovery) IJ000903: Error creating Subject for crash recovery: java:jboss/datasources/RHQDS (null): java.lang.NullPointerException
	at org.jboss.security.plugins.JBossSecuritySubjectFactory.createSubject(JBossSecuritySubjectFactory.java:83)
	at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl$1.run(XAResourceRecoveryImpl.java:300)
	at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl$1.run(XAResourceRecoveryImpl.java:261)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_11]
	at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.getSubject(XAResourceRecoveryImpl.java:260)
	at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.getXAResources(XAResourceRecoveryImpl.java:165)
	at com.arjuna.ats.internal.jbossatx.jta.XAResourceRecoveryHelperWrapper.getXAResources(XAResourceRecoveryHelperWrapper.java:51) [jbossjts-integration-4.17.18.Final.jar:4.17.18.Final]
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:516) [jbossjts-jacorb-4.17.18.Final.jar:4.17.18.Final]
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:182) [jbossjts-jacorb-4.17.18.Final.jar:4.17.18.Final]
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb-4.17.18.Final.jar:4.17.18.Final]
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.18.Final.jar:4.17.18.Final]
Comment 13 John Mazzitelli 2014-03-28 14:59:48 UTC 
OK, finally got things started on DR6 and I see that we still have https://issues.jboss.org/browse/WFLY-3124 being a problem. Here's what we get (this error causes problems, not least of which is the agents can't talk to the servers since JMX via JBoss/Remoting is involved in such communications):

Caused by: java.util.NoSuchElementException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:897) [rt.jar:1.7.0_11]
	at java.util.HashMap$KeyIterator.next(HashMap.java:928) [rt.jar:1.7.0_11]
	at org.jboss.as.jmx.PluggableMBeanServerImpl$LogAction.getCallerUserId(PluggableMBeanServerImpl.java:1249)
	at org.jboss.as.jmx.PluggableMBeanServerImpl$LogAction.doLog(PluggableMBeanServerImpl.java:1232)
Comment 14 Kabir Khan 2014-04-04 15:48:23 UTC 
THis is part of https://bugzilla.redhat.com/show_bug.cgi?id=1081675 which was done for DR6
Comment 15 John Mazzitelli 2014-04-15 14:25:32 UTC 
Just tested with the latest JBEAP-6.3.0.ER1 build from April 4, 2014 and still getting this error, which is a blocker for JON 3.3. This error causes, among other problems, agents unable to talk to servers.
Comment 16 John Mazzitelli 2014-04-15 14:26:50 UTC 
(In reply to John Mazzitelli from comment #15)
> Just tested with the latest JBEAP-6.3.0.ER1 build from April 4, 2014 and
> still getting this error, which is a blocker for JON 3.3. This error causes,
> among other problems, agents unable to talk to servers.

Sorry - forgot to be clear - the error we are still getting is this: https://issues.jboss.org/browse/WFLY-3124
Comment 17 John Mazzitelli 2014-04-15 14:34:47 UTC 
(In reply to John Mazzitelli from comment #16)
> (In reply to John Mazzitelli from comment #15)
> > Just tested with the latest JBEAP-6.3.0.ER1 build from April 4, 2014 and
> > still getting this error, which is a blocker for JON 3.3. This error causes,
> > among other problems, agents unable to talk to servers.
> 
> Sorry - forgot to be clear - the error we are still getting is this:
> https://issues.jboss.org/browse/WFLY-3124

For backporting that WildFly issue into EAP, see bug #1082072
Comment 18 John Mazzitelli 2014-05-08 18:22:47 UTC 
running ER4 and the problem from comment #10 has come back. The solution in comment #11 is not working anymore.
Comment 19 John Mazzitelli 2014-05-08 18:45:26 UTC 
(In reply to John Mazzitelli from comment #18)
> running ER4 and the problem from comment #10 has come back. The solution in
> comment #11 is not working anymore.

for the record, this is related to bug #1080826
Comment 20 John Mazzitelli 2014-05-08 19:13:44 UTC 
as unbelievable as it sounds - my test war that replicated the problem? it works now if I just put it in "deployments/" of ER4 without passing anything extra as JAVA_OPTS, I just run standalone.sh and it works.

JON no longer works, with or without that extra property.

Same exception as before it seems:

Caused by: java.lang.ClassNotFoundException: org.rhq.enterprise.server.core.service.ManagementService$GetModelControllerService from [Module "deployment.rhq.ear.rhq-server.jar:main" from Service Module Loader]
	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389) [jboss-modules.jar:1.3.3.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134) [jboss-modules.jar:1.3.3.Final-redhat-1]
	... 8 more
Comment 21 John Mazzitelli 2014-05-08 19:47:00 UTC 
this is very weird. If I put the test war in the JON's EAP deployments/ directory, it fails again. I need to double check what I'm doing.


This is the version of EAP that JON is using, it should be the 6.3.ER4 one:

15:44:25,203 INFO  [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final-redhat-1
15:44:25,494 INFO  [org.jboss.msc] (main) JBoss MSC version 1.1.5.Final-redhat-1
15:44:25,615 INFO  [org.jboss.as] (MSC service thread 1-7) JBAS015899: JBoss EAP 6.3.0.Beta2 (AS 7.4.0.Final-redhat-12) starting
Comment 22 John Mazzitelli 2014-05-08 21:17:00 UTC 
ok, looks like the problem is that now in ER4 this sysprop "-Djava.security.manager" is causing problems. If I take that out, I don't get the error and the RHQ server can again be installed and run.

Need to ask Lukas about this - I believe he was the one that added in the security manager stuff. Hopefully, we can take out this sysprop from our server's startup scripts. But I don't know - its here for a reason.
Comment 23 John Mazzitelli 2014-05-08 21:23:02 UTC 
JON is in a worse way with ER4 compared to ER2. At least before I could get to the login page, and log in and see most of the GUI. Now, I can't even do that. In addition, looks like the agents still can't talk to the servers. But I don't know why - perhaps the same reason why I can't get to the UI? In any event, JON Server is not working in ER4 - I can't even confirm if the WLDFLY bug we were waiting to be fixed is actually fixed for us.
Comment 24 John Mazzitelli 2014-05-08 22:13:05 UTC 
here's another weird thing I'm seeing now - the server just "stops" in the middle of starting during installation. For example:

18:09:49,956 INFO  [org.jboss.as.remoting] (MSC service thread 1-8) JBAS017100: Listening on 127.0.0.1:4447
RHQ Server                     (pid 11107  ) is starting
18:09:51,092 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 58) HQ221020: Started Netty Acceptor version 3.6.7.Final-redhat-1-3c0a3ac 127.0.0.1:5455 for CORE protocol
18:09:51,094 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 58) HQ221020: Started Netty Acceptor version 3.6.

Notice that last log message just stopped in the middle! Its like the logging output halted in the middle of spitting out a message. Its been doing this consistently, but never in the same place (that is, the server just "stops" during startup at install time, but it stops at different places different times I run).
Comment 25 John Mazzitelli 2014-05-08 22:16:32 UTC 
(In reply to John Mazzitelli from comment #24)
> here's another weird thing I'm seeing now - the server just "stops" in the
> middle of starting during installation. For example:

I need to check, I think it might be because I ran rhqctl install with "RHQ_SERVER_DEBUG" set to true.
Comment 26 John Mazzitelli 2014-05-09 02:20:47 UTC 
ok, after some fits and starts, ER4 is running in better shape than previous ERs. I have an agent registered. Looks like the WFLY-3124 issue is fixed for us. However, I need to remove -Djava.security.manager from the rhq-server.sh script for it to work. I don't know what breaks if we do this. Need lukas' input.

There is some errors in the server log that needs to be figured out. I don't know what went bad. But looks like hibernate connection issues:

22:11:50,014 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-3) SQL Error: 0, SQLState: null
22:11:50,015 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-3) javax.resource.ResourceException: IJ000456: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3053b8b7[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@48840f2a connection handles=0 lastUse=1399601509911 trackByTx=true pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@4dba1c9b pool internal context=SemaphoreArrayListManagedConnectionPool@60faaf19[pool=RHQDS] xaResource=XAResourceWrapperImpl@7a7b6670[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@48840f2a pad=false overrideRmValue=null productName=PostgreSQL productVersion=9.0.5 jndiName=java:jboss/datasources/RHQDS] txSync=TxSync@1964849907{tx=TransactionImple < ac, BasicAction: 0:ffffc0a80102:561cd2d9:536c38a8:60f5 status: ActionStatus.RUNNING > wasTrackByTx=true enlisted=true}]
22:11:50,019 ERROR [org.jboss.as.ejb3] (RHQScheduler_Worker-3) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
22:11:50,019 ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-3) JBAS014134: EJB Invocation failed on component ResourceTypeManagerBean for method public abstract org.rhq.core.domain.util.PageList org.rhq.enterprise.server.resource.ResourceTypeManagerRemote.findResourceTypesByCriteria(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.criteria.ResourceTypeCriteria): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection

There is VERY long stack traces in here. I won't post it all but here's some relevant portions:

The last "caused by" in the cause-chain:

Caused by: javax.resource.ResourceException: No matching credentials in Subject!
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:965)
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.checkIdentity(BaseWrapperManagedConnection.java:845)
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.getConnection(BaseWrapperManagedConnection.java:466)
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:497)

Another error:

22:11:50,038 ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-3) JBAS014134: EJB Invocation failed on component InventoryManagerBean for method public abstract java.util.List org.rhq.enterprise.server.inventory.InventoryManagerLocal.getDeletedTypes(): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection

...
	at org.rhq.enterprise.server.inventory.InventoryManagerLocal$$$view35.getDeletedTypes(Unknown Source) [rhq-server.jar:4.12.0-SNAPSHOT]
	at org.rhq.enterprise.server.scheduler.jobs.PurgeResourceTypesJob.executeJobCode(PurgeResourceTypesJob.java:52) [rhq-server.jar:4.12.0-SNAPSHOT]
	at org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob.execute(AbstractStatefulJob.java:48) [rhq-server.jar:4.12.0-SNAPSHOT]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-1.6.5.jar:1.6.5]
...
Caused by: javax.resource.ResourceException: No matching credentials in Subject!
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:965)
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.checkIdentity(BaseWrapperManagedConnection.java:845)
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.getConnection(BaseWrapperManagedConnection.java:466)
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:497)
Comment 27 Lukas Krejci 2014-05-09 09:10:40 UTC 
A security manager is needed so that we can run CLI scripts without the fear of them shutting down the server or circumventing authorization by going directly to the database (by looking up entity manager in the JNDI), etc.

So if we can configure the security manager WFLY now seems to have (as far as I know) instead of using the default one initialized by the -Djava.security.manager then we should be fine.

The test is simple. Create an alert definition with a CLI notification, whose js script will simply run:

java.lang.System.exit(0);

If the server shuts down once the alert fires, we have a problem. If the notification merely fails with a security exception, we're fine.
Comment 28 John Mazzitelli 2014-05-12 14:31:56 UTC 
Created attachment 894738 [details]
policy file to use via -Djava.security.policy==

To test on EAP 6.3. ER4, run with sys props "-Djava.security.manager" and "-Djava.security.policy==<the attached policy file>" (notice the double-equals is not a typo when setting policy file).

The policy file is simply this:

grant {
    permission java.security.AllPermission;
};
Comment 29 John Mazzitelli 2014-05-12 14:58:46 UTC 
logged the latest issue in https://bugzilla.redhat.com/show_bug.cgi?id=1082467#c5
Comment 30 John Mazzitelli 2014-05-16 15:13:57 UTC 
Wrote up the latest blocker here: bug #1098577
Comment 31 John Mazzitelli 2014-05-27 21:07:53 UTC 
Running a local build of EAP 6.3 so I can pick up the latest patches I need, I run but get the following exception many times as Quartz attempts to obtain our no-tx datasource:

java.sql.SQLException: javax.resource.ResourceException: IJ000456: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@508c0593[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@18c21f35 connection handles=0 lastUse=1401224473867 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@28d1bc4 pool internal context=SemaphoreArrayListManagedConnectionPool@653277f1[pool=NoTxRHQDS]]
     at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
     at org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:160)
     at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:112)
     at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:164)
     at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3750)
     at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3722)
     at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2958)
     at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1727)
     at org.quartz.core.JobRunShell.run(JobRunShell.java:273)
     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
 Caused by: javax.resource.ResourceException: IJ000456: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@508c0593[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@18c21f35 connection handles=0 lastUse=1401224473867 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@28d1bc4 pool internal context=SemaphoreArrayListManagedConnectionPool@653277f1[pool=NoTxRHQDS]]
     at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:511)
     at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:143)
     ... 9 more
 Caused by: javax.resource.ResourceException: No matching credentials in Subject!
     at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:965)
     at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.checkIdentity(BaseWrapperManagedConnection.java:845)
     at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.getConnection(BaseWrapperManagedConnection.java:466)
     at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:497)
     ... 10 more
Comment 32 John Mazzitelli 2014-05-28 19:10:43 UTC 
(In reply to John Mazzitelli from comment #31)
> Running a local build of EAP 6.3 so I can pick up the latest patches I need,
> I run but get the following exception many times as Quartz attempts to
> obtain our no-tx datasource:

This has been written up as bug #1102332
Comment 33 John Mazzitelli 2014-05-30 20:53:47 UTC 
ok, pushed two changes - one, turning off java.security.manager, second (git commit 4e8484e) the installer now creates two security domains - one per DB datasource. That fixes the problems we hit up to now.

Now when I run, I'm getting more other problems

1) when I run ANY operation, I get back no results, no matter which op, or which resource.

2) the left-hand tree nav sometimes fails to render and I get a globally caught exception. The stack traces look different, I saw two different stacks.

---the first error---

 columnNumber: 2: this$static is null
   at Unknown.org_rhq_core_domain_resource_ResourceType_$getId__Lorg_rhq_core_domain_resource_ResourceType_2I(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository_$loadResourceTypes__Lorg_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository_2Ljava_util_List_2Ljava_util_EnumSet_2Lorg_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository$ResourceTypeLoadedCallback_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_$processIncomingData__Lorg_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_2Ljava_util_List_2Lcom_smartgwt_client_data_DSResponse_2Ljava_lang_String_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_$executeFetch__Lorg_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_2Ljava_lang_String_2Lcom_smartgwt_client_data_DSRequest_2Lcom_smartgwt_client_data_DSResponse_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_transformRequest__Lcom_smartgwt_client_data_DSRequest_2Ljava_lang_Object_2(Unknown Source)
   at Unknown.com_smartgwt_client_data_DataSource_onInit__V/self.transformRequest<(Unknown Source)
   at Unknown.com_google_gwt_core_client_impl_Impl_apply__Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2(Unknown Source)
   at Unknown.com_google_gwt_core_client_impl_Impl_entry0__Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2(Unknown Source)
   at Unknown.anonymous(Unknown Source)
   at Unknown.isc_DataSource_getServiceInputs(Unknown Source)
   at Unknown.isc_DataSource_sendDSRequest(Unknown Source)
   at Unknown.isc_DataSource_performDSOperation(Unknown Source)
   at Unknown.isc_DataSource_fetchData(Unknown Source)
   at Unknown.isc_ResultTree_loadChildren(Unknown Source)
   at Unknown.isc_Tree_changeDataVisibility(Unknown Source)
   at Unknown.isc_Tree_openFolder(Unknown Source)
   at Unknown.isc_Tree_setRoot(Unknown Source)
   at Unknown.isc_Tree_init(Unknown Source)
   at Unknown.isc_c_Class_invokeSuper(Unknown Source)
   at Unknown.isc_ResultTree_init(Unknown Source)
   at Unknown.isc_Class_completeCreation(Unknown Source)
   at Unknown.isc_c_Class_create(Unknown Source)
   at Unknown.isc_Canvas_createResultTree(Unknown Source)
   at Unknown.isc_TreeGrid_createDataModel(Unknown Source)
   at Unknown.isc_Canvas_filterWithCriteria(Unknown Source)
   at Unknown.isc_Canvas__filter(Unknown Source)
   at Unknown.isc_c_Class_invokeSuper(Unknown Source)
   at Unknown.isc_c_Class_Super(Unknown Source)
   at Unknown.isc_ListGrid__filter(Unknown Source)
   at Unknown.isc_Canvas_fetchData(Unknown Source)
   at Unknown.com_smartgwt_client_widgets_tree_TreeGrid_$fetchData__Lcom_smartgwt_client_widgets_tree_TreeGrid_2Lcom_smartgwt_client_data_Criteria_2Lcom_smartgwt_client_data_DSCallback_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeView$14_$onSuccess__Lorg_rhq_coregui_client_inventory_resource_detail_ResourceTreeView$14_2Ljava_util_List_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeView$14_onSuccess__Ljava_lang_Object_2V(Unknown Source)
   at Unknown.com_google_gwt_user_client_rpc_impl_RequestCallbackAdapter_onResponseReceived__Lcom_google_gwt_http_client_Request_2Lcom_google_gwt_http_client_Response_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_util_rpc_TrackingRequestCallback_onResponseReceived__Lcom_google_gwt_http_client_Request_2Lcom_google_gwt_http_client_Response_2V(Unknown Source)
   at Unknown.com_google_gwt_http_client_Request_$fireOnResponseReceived__Lcom_google_gwt_http_client_Request_2Lcom_google_gwt_http_client_RequestCallback_2V(Unknown Source)
   at Unknown.com_google_gwt_http_client_RequestBuilder$1_onReadyStateChange__Lcom_google_gwt_xhr_client_XMLHttpRequest_2V(Unknown Source)
   at Unknown.this$static.onreadystatechange<(Unknown Source)




---and the second error---



 columnNumber: 2: this$static is null
   at Unknown.org_rhq_core_domain_resource_ResourceType_$getId__Lorg_rhq_core_domain_resource_ResourceType_2I(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository_$loadResourceTypes__Lorg_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository_2Ljava_util_List_2Ljava_util_EnumSet_2Lorg_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository$ResourceTypeLoadedCallback_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeView$14_$onSuccess__Lorg_rhq_coregui_client_inventory_resource_detail_ResourceTreeView$14_2Ljava_util_List_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeView$14_onSuccess__Ljava_lang_Object_2V(Unknown Source)
   at Unknown.com_google_gwt_user_client_rpc_impl_RequestCallbackAdapter_onResponseReceived__Lcom_google_gwt_http_client_Request_2Lcom_google_gwt_http_client_Response_2V(Unknown Source)
   at Unknown.org_rhq_coregui_client_util_rpc_TrackingRequestCallback_onResponseReceived__Lcom_google_gwt_http_client_Request_2Lcom_google_gwt_http_client_Response_2V(Unknown Source)
   at Unknown.com_google_gwt_http_client_Request_$fireOnResponseReceived__Lcom_google_gwt_http_client_Request_2Lcom_google_gwt_http_client_RequestCallback_2V(Unknown Source)
   at Unknown.com_google_gwt_http_client_RequestBuilder$1_onReadyStateChange__Lcom_google_gwt_xhr_client_XMLHttpRequest_2V(Unknown Source)
   at Unknown.this$static.onreadystatechange<(Unknown Source)
Comment 34 John Mazzitelli 2014-05-30 21:18:12 UTC 
I thought maybe it was because I was using an older version of FireFox, but Chrome had the same problems. Here's the exception when the left-hand tree failed to render (note, only sometimes does the left-hand tree fail to render):

com.google.gwt.core.client.JavaScriptException:(TypeError) : Cannot read property 'org_rhq_core_domain_resource_ResourceType_id' of null
--- STACK TRACE FOLLOWS ---
(TypeError) : Cannot read property 'org_rhq_core_domain_resource_ResourceType_id' of null
   at Unknown.org_rhq_core_domain_resource_ResourceType_$getId__Lorg_rhq_core_domain_resource_ResourceType_2I(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@21)
   at Unknown.org_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository_$loadResourceTypes__Lorg_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository_2Ljava_util_List_2Ljava_util_EnumSet_2Lorg_rhq_coregui_client_inventory_resource_type_ResourceTypeRepository$ResourceTypeLoadedCallback_2V(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@86)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_$processIncomingData__Lorg_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_2Ljava_util_List_2Lcom_smartgwt_client_data_DSResponse_2Ljava_lang_String_2V(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@3)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_$executeFetch__Lorg_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_2Ljava_lang_String_2Lcom_smartgwt_client_data_DSRequest_2Lcom_smartgwt_client_data_DSResponse_2V(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@7)
   at Unknown.org_rhq_coregui_client_inventory_resource_detail_ResourceTreeDatasource_transformRequest__Lcom_smartgwt_client_data_DSRequest_2Ljava_lang_Object_2(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@201)
   at Unknown.<anonymous>(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@21)
   at Unknown.com_google_gwt_core_client_impl_Impl_apply__Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@23)
   at Unknown.com_google_gwt_core_client_impl_Impl_entry0__Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2Ljava_lang_Object_2(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@16)
   at Unknown.transformRequest(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/D8F15341A72C5D60D95EAB092751BD9C.cache.html@14)
   at Unknown.isc_DataSource_getServiceInputs(http://localhost:7080/coregui/org.rhq.coregui.CoreGUI/sc/modules/ISC_DataBinding.js@13)
Comment 35 John Mazzitelli 2014-06-02 19:43:30 UTC 
one of the exceptions is because GWT is trying to load the tree, and the first resource getting processed doesn't have the resource type loaded (in fact, it has very little data).

org.rhq.coregui.client.inventory.resource.detail.ResourceTreeView.loadTree(int, boolean, AsyncCallback<Void>)

eventually calls into

org.rhq.coregui.client.inventory.resource.type.ResourceTypeRepository.loadResourceTypes(List<Resource>, EnumSet<MetadataType>, ResourceTypeLoadedCallback)

but the List<Resource> argument's first item in the list doesn't have the resource type (nor much of anything).

The toString is:

Resource[id=10001, uuid=null, type=<null>, key=null, name=null, parent=<null>]

and the Inventory Status is NEW.

What up with that?
Comment 36 John Mazzitelli 2014-06-02 20:31:51 UTC 
(In reply to John Mazzitelli from comment #35)
It all boils down to the new Hibernate in EAP 6.3 and how the new javassist proxy names are different. Our HibernateDetachUtility would search for "javassist" in the HibernateProxy class name and it if matches, we'd do things to scrub it. However, the names of those internal proxies are now different - for example:

org.rhq.core.domain.resource.Resource_$$_jvsta24_21

I think we need to be looking for "jvsta" now, not "javassist".
Comment 37 John Mazzitelli 2014-06-02 20:44:56 UTC 
(In reply to John Mazzitelli from comment #36)
> (In reply to John Mazzitelli from comment #35)
> It all boils down to the new Hibernate in EAP 6.3 and how the new javassist
> proxy names are different. Our HibernateDetachUtility would search for
> "javassist" in the HibernateProxy class name and it if matches, we'd do
> things to scrub it. However, the names of those internal proxies are now
> different - for example:
> 
> org.rhq.core.domain.resource.Resource_$$_jvsta24_21
> 
> I think we need to be looking for "jvsta" now, not "javassist".

OK, here's the actual git commit in Javassist that changed the naming:

https://github.com/jboss-javassist/javassist/commit/3358dc269352383eb5e9c186d24ff4a292eeb637

 +        private final String sep = "_$$_jvst" + Integer.toHexString(this.hashCode() & 0xfff) + "_";

versus

-        return classname + "_$$_javassist_" + counter++;

So we need to look for "jvst" not "javassist".
Comment 38 John Mazzitelli 2014-06-02 20:52:04 UTC 
git commit to the feature branch: 811544eb5a7bfce6243e9c7e7ee0ce874da4146e

We now longer look for "javassist" in the proxy name, but now we are looking for "jvst".

This fixes the latest two issues I saw - the left hand tree is now rendered successfully and operation results is showing me correct data.
Comment 39 John Mazzitelli 2014-06-19 12:50:05 UTC 
The work here is largely done now, in the rebase branch of RHQ repo.

I also tried it built against 6.3. ER7 build, and it works. It works because of the bugs that are still outstanding in that version, we worked around them (by turning off the security manager and installing a security domain per datasource).

I will try to see if the build works against the 6.3 alpha release (though I'm pessimistic that it will work, I think that might be too early a build - I think there were some bugs that we couldn't workaround in that version, but I'll test to see if we luck out and it works).
Comment 40 John Mazzitelli 2014-06-20 02:21:56 UTC 
I have this running for the most part on EAP 6.3 Alpha - all unit tests and integration tests pass - I got a full, clean "mvn install" build to succeed.

Two things to mention:

1) rhqctl install --start doesn't seem to work. I get some odd NIO exceptions and neither the agent nor the GUI work (both don't work probably because they go through the same web connector). I get an exception similar to this - and it could be the same problem: https://bugzilla.redhat.com/show_bug.cgi?id=1090291 (note that BZ's title seem to indicate its only a problem on  Windows, but I see it on Fedora.)

WORKAROUND: Install via "rhqctl install", then start separately via "rhqctl start"

2) rest-api - all but 4 tests pass. These fail:

Failed tests:   testCreatePackageBasedResource(org.rhq.modules.integrationTests.restApi.ContentTest): 1 expectation failed.
  org.rhq.modules.integrationTests.restApi.PluginsTest: 1 expectation failed.
  testCreateResourceRegularChildAndRemove(org.rhq.modules.integrationTests.restApi.ResourcesTest): 1 expectation failed.
  testCreateResourceRegularChildEmptyConfigAndRemove(org.rhq.modules.integrationTests.restApi.ResourcesTest): 1 expectation failed.

Tests in error: 
  org.rhq.modules.integrationTests.restApi.PluginsTest: Index: 0, Size: 0

I believe one of those is a known failure (and fails in master) - I think the PluginsTest one. The others did not fail when I built on the latest 6.x build.
Comment 41 John Mazzitelli 2014-06-26 16:24:51 UTC 
commit to master: d2427f017e3769945e7767da6b11a072532bc5fa

this is the code to rebase RHQ Server on EAP 6.3.Alpha.
Comment 42 JBoss JIRA Server 2014-06-27 08:26:39 UTC 
Heiko Rupp <hrupp> updated the status of jira JON3-9 to Resolved
Comment 43 John Mazzitelli 2014-07-08 18:00:07 UTC 
this is all done and in master
Comment 44 Simeon Pinder 2014-07-31 15:51:47 UTC 
Moving to ON_QA as available to test with brew build of DR01: https://brewweb.devel.redhat.com//buildinfo?buildID=373993