Bug 1127272

Summary: Exception in server.log when installing JON server via 'rhqctl install --start'
Product: [JBoss] JBoss Operations Network Reporter: Filip Brychta <fbrychta>
Component: InstallerAssignee: Jay Shaughnessy <jshaughn>
Status: CLOSED CURRENTRELEASE QA Contact: Filip Brychta <fbrychta>
Severity: high Docs Contact:
Priority: unspecified    
Version: JON 3.3.0CC: fbrychta, hrupp, jcosta, jshaughn, mazz, mfoley, mmahoney
Target Milestone: ER04   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-11 13:59:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1032641    
Bug Blocks:    
Attachments:
Description Flags
server.log none

Description Filip Brychta 2014-08-06 13:54:38 UTC
Description of problem:
If the '--start' option i used while installing the server using rhqctl script, there are error messages logged in the server.log file.

Version-Release number of selected component (if applicable):
Version :	
3.3.0.DR01
Build Number :	
6468454:dda0a47

How reproducible:
Always

Steps to Reproduce:
1. Install the server by invoking 'rhqctl install --start'
2. Log in
3. Check the contents of server.log

Actual results:
It is possible to log in but following exception is thrown to server.log:
08:16:45,548 ERROR [org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl] (Periodic Recovery) IJ000906: Error during crash recovery: java:jboss/datasources/RHQDS (Could not create connection): javax.resource.ResourceException: Could not create connection
	at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:514)
	at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory$1.run(XAManagedConnectionFactory.java:410)
	at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory$1.run(XAManagedConnectionFactory.java:407)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_45]
	at javax.security.auth.Subject.doAs(Subject.java:415) [rt.jar:1.7.0_45]
	at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:406)
	at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.open(XAResourceRecoveryImpl.java:343)
	at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.getXAResources(XAResourceRecoveryImpl.java:170)
	at com.arjuna.ats.internal.jbossatx.jta.XAResourceRecoveryHelperWrapper.getXAResources(XAResourceRecoveryHelperWrapper.java:51) [jbossjts-integration-4.17.21.Final-redhat-2.jar:4.17.21.Final-redhat-2]
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:516) [jbossjts-jacorb-4.17.21.Final-redhat-2.jar:4.17.21.Final-redhat-2]
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:182) [jbossjts-jacorb-4.17.21.Final-redhat-2.jar:4.17.21.Final-redhat-2]
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb-4.17.21.Final-redhat-2.jar:4.17.21.Final-redhat-2]
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.21.Final-redhat-2.jar:4.17.21.Final-redhat-2]
Caused by: java.lang.IllegalStateException
	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
	at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.driverClassLoader(AbstractDataSourceService.java:237)
	at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.access$200(AbstractDataSourceService.java:89)
	at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer$1.getClassLoader(AbstractDataSourceService.java:402)
	at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXADataSource(XAManagedConnectionFactory.java:634)
	at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:497)
	... 12 more

Expected results:
No exception

Comment 2 Filip Brychta 2014-08-07 07:03:31 UTC
*** Bug 1127354 has been marked as a duplicate of this bug. ***

Comment 3 Heiko W. Rupp 2014-08-26 12:42:14 UTC
Full server log up until this point or it did not happen :-)

Comment 4 Filip Brychta 2014-08-26 13:04:15 UTC
Created attachment 930952 [details]
server.log

It did happen:)

Comment 5 John Mazzitelli 2014-08-29 20:23:38 UTC
I have seen this many times, never knew what it was. Has nothing to do with metric data or things like that. I've seen this on clean installs with nothing in inventory yet. IIRC, I see thing after I control-c a server, unzip another distro, and restall it again pointing to the same DB. I really don't know why this XA error is happening, but try to install, kill the server, and re-install and see if you get the error.

Comment 6 Filip Brychta 2014-09-01 09:58:54 UTC
This exception is visible in server.log for both the clean installation (with empty db) and second installation (using existing db). In other words I didn't find any scenario where this issues is NOT visible.

Comment 7 Jay Shaughnessy 2014-09-09 17:49:29 UTC
Maybe relevant: https://developer.jboss.org/thread/195534.  but I'm not sure we want to play with the CCM setting.

This seems like it may be an EAP issue: https://developer.jboss.org/thread/241721.  In this the issue was the same and went away with Wildfly 8.1.

Another option is to filter away these messages. Setting to ER04 for now, but may push to post-ga.

Comment 8 Juraci Paixão Kröhling 2014-09-12 13:26:36 UTC
While debugging this issue, I found out that the value for the driver name isn't being injected all the time, thus failing at the "InjectedValue" with an InvalidStateException. As the configuration on RHQ's side looks sane, I talked with the IronJacamar team and they pointed out to me that it's not a configuration issue at RHQ's side, but an existing bug on the AS: https://bugzilla.redhat.com/show_bug.cgi?id=1032641

In the end, I'm not sure if there's anything to do at the RHQ's side.

Comment 11 Jay Shaughnessy 2014-09-24 15:25:31 UTC
Pulled this back to 3.3.0 because I don't think we'll be able to do anything about this in a CP.  Adding a filter until we layer on an EAP version with the fix.

Comment 12 Jay Shaughnessy 2014-09-24 18:51:33 UTC
commit 2c44cde5c5001edf5cf8b1ebcbc1fa98d59cbd91
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Sep 24 13:43:32 2014 -0400

    Add EAP-level log filters for messages we can't avoid and don't want to see.



release/jon3.3.x commit 1b241d7a28f65737762e98250cf8b18f18c1377c
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Sep 24 13:43:32 2014 -0400

    (cherry picked from commit 2c44cde5c5001edf5cf8b1ebcbc1fa98d59cbd91)
    Signed-off-by: Jay Shaughnessy <jshaughn>

Comment 13 Simeon Pinder 2014-10-01 21:33:08 UTC
Moving to ON_QA as available for test with build:
https://brewweb.devel.redhat.com/buildinfo?buildID=388959

Comment 14 Filip Brychta 2014-10-02 10:31:27 UTC
Verified on
Version :	
3.3.0.ER04
Build Number :	
99d2107:d7c537e