Bug 1117801

Summary: Server crashes after installation via 'rhqctl install --start' (EAP 6.3 alpha issue)
Product: [JBoss] JBoss Operations Network Reporter: Jan Bednarik <jbednari>
Component: InstallerAssignee: Simeon Pinder <spinder>
Status: CLOSED CURRENTRELEASE QA Contact: Filip Brychta <fbrychta>
Severity: high Docs Contact:
Priority: high    
Version: JON 3.2.2CC: fbrychta, jshaughn, mazz, mkoci
Target Milestone: DR01   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1148825 (view as bug list) Environment:
Last Closed: 2014-12-11 14:00:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1148825    
Attachments:
Description Flags
Server log file none

Description Jan Bednarik 2014-07-09 12:22:56 UTC
Created attachment 916755 [details]
Server log file

Description of problem:
If the '--start' option i used while installing the server using rhqctl script, the server crashes. All storage node, agent and storage seems to be running (the output of 'rhqctl status' says so) but there are error messages logged in the server.log file and the web UI does not load.

The exception logged in server.log file (it is logged multiple time):

08:03:31,516 ERROR [org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl] (Periodic Recovery) IJ000906: Error during crash recovery: java:jboss/datasources/RHQDS (Could not create connection): javax.resource.ResourceException: Could not create connection
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:461)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:398)
        at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.open(XAResourceRecoveryImpl.java:343)
        at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.getXAResources(XAResourceRecoveryImpl.java:170)
        at com.arjuna.ats.internal.jbossatx.jta.XAResourceRecoveryHelperWrapper.getXAResources(XAResourceRecoveryHelperWrapper.java:51) [jbossjts-integration-4.17.17.Final.jar:4.17.17.Final]
        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:516) [jbossjts-jacorb-4.17.17.Final.jar:4.17.17.Final]
        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:182) [jbossjts-jacorb-4.17.17.Final.jar:4.17.17.Final]
        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb-4.17.17.Final.jar:4.17.17.Final]
        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.17.Final.jar:4.17.17.Final]
Caused by: java.lang.IllegalStateException
        at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
        at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.driverClassLoader(AbstractDataSourceService.java:237)
        at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.access$200(AbstractDataSourceService.java:89)
        at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer$1.getClassLoader(AbstractDataSourceService.java:402)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXADataSource(XAManagedConnectionFactory.java:581)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:444)
        ... 8 more

Version-Release number of selected component (if applicable):
RHQ 4.12-SNAPSHOT (7bf3544)

The same problem occurs both on RHEL 6.5 and Windows Server 2008.

How reproducible:
always

Steps to Reproduce:
1. Install the server by invoking 'rhqctl install --start'
2. Try to load the web UI.
3. Check the contents of server.log

Actual results:
The web UI does not load and the error messages are logged.

Expected results:
The web UI loads and no error messages are logged.

Additional info:
If the server is installed using only 'rhqctl install' and then started using 'rhqctl start', everything works fine.

Comment 1 John Mazzitelli 2014-07-09 13:23:04 UTC
This is already mentioned in the Release Notes for 4.12. This problem will go away when we upgrade to a newer EAP version (this problem goes away, for example, when running on top of EAP 6.3.GA).

Comment 2 Jay Shaughnessy 2014-07-28 19:17:40 UTC
Since Simeon is going to flip the EAP 6.3 GA switch when it becomes available, I'm assigning this to him.  It should be able to go straight to MODIFIED at that time, assuming mazz is correct in his statement above.

Comment 3 Simeon Pinder 2014-08-05 13:40:13 UTC
Missed this in initial mass bz update to ON_QA.  Moving this to ON_QA for DR01 ton confirm issue no longer there with this rebased version.

Comment 4 Filip Brychta 2014-08-06 12:07:52 UTC
The exception is still visible in server.log in version:
Version :	
3.3.0.DR01
Build Number :	
6468454:dda0a47

but the UI is accessible and I'm able to log in.

Comment 5 John Mazzitelli 2014-08-06 13:17:43 UTC
(In reply to Filip Brychta from comment #4)
> The exception is still visible in server.log in version:
> Version :	
> 3.3.0.DR01
> Build Number :	
> 6468454:dda0a47
> 
> but the UI is accessible and I'm able to log in.

That exception is different from the problem of not being able to use --start. I think a different BZ should be written up on that.

The issue we needed to fix in this BZ  what as the description states: " Server crashes after installation via 'rhqctl install --start'" Using --start, and you could never start the server successfully during install, the server would actually crash at startup and you could not log in (the reason was IIRC some sockets/connectors failed to initialize after a reload - this XA exception is different).

The fact that you can log in shows that the connector problem was solved and the server doesn't crash. It might get an exception in the log, but that is not crashing.

I would close this BZ as fixed because now you can use --start and be able to log in.

I would open another BZ and log that exception. That is also an issue that we've seen before (and I've heard others using EAP say they see the same thing) but as far as I know, the server is still working.

Comment 6 Filip Brychta 2014-08-06 13:56:25 UTC
Based on comment 5, marking this bz as verified. New bz 1127272 created.