Bug 1087422

Summary: Unable to start (autodetected) EAP 5.3 via JON after being stopped via JON on Solaris 10 both Intel and sparc 64 bit architecture
Product: [JBoss] JBoss Operations Network Reporter: Radim Hatlapatka <rhatlapa>
Component: Plugin -- JBoss EAP 5Assignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED WORKSFORME QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: JON 3.2CC: loleary, myarboro, rhatlapa, tsegismo
Target Milestone: ---   
Target Release: JON 3.3.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1119859 (view as bug list) Environment:
Last Closed: 2015-01-09 23:52:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1119859    
Attachments:
Description Flags
agent log, please note, that the exceptions are in time of previous test (manualy importing secured EAP 5) none

Description Radim Hatlapatka 2014-04-14 10:19:13 UTC
Created attachment 886094 [details]
agent log, please note, that the exceptions are in time of previous test (manualy importing secured EAP 5)

Description of problem:
Unable to start EAP 5.3 (imported via discovery queue & autodetection) via JON on Solaris 10 Sparc (64bit Oracle JDK) and Windows (all combinations)

Version-Release number of selected component (if applicable): JON 3.2.0 vs EAP 5.3.0.ER3


How reproducible: always


Steps to Reproduce:
1. import running EAP 5.3 server to JON (on one of the impacted platforms)
2. shutdown the EAP server via JON
3. try to start the EAP server via JON

Actual results:
java.lang.Exception: The server failed to start: at org.rhq.core.pc.operation.OperationInvocation.run(OperationInvocation.java:278) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 

Expected results:
EAP server starts


Additional info:
no log information found on JON server side, and no information are in EAP 5 logs (the EAP 5.3 start up process is not even started.)

The same works just fine with EAP 5.2

I am also attaching log of JON agent, where in comparison with EAP 5.2 there is one additional exception from previous test.
java.lang.NullPointerException
  at org.rhq.plugins.jbossas5.util.ManagedComponentUtils.getSimplePropertyValue(ManagedComponentUtils.java:67)
  at org.rhq.plugins.jbossas5.ApplicationServerComponent.getMetric(ApplicationServerComponent.java:626)
  at org.rhq.plugins.jbossas5.ApplicationServerComponent.getValues(ApplicationServerComponent.java:259)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  at java.lang.Thread.run(Thread.java:662)

Comment 1 Radim Hatlapatka 2014-04-14 10:31:28 UTC
The same issue I am getting on Solaris 10 Intel architecture (only with Oracle JDK 1.6 64bit)

Comment 2 Thomas Segismont 2014-04-24 15:41:59 UTC
Could not reproduce with EAP 5.3.ER3, JON 3.2GA on Windows 2008 server

EAP 6.3 is auto discovered and the stop and start operations work as expected.

Comment 3 Radim Hatlapatka 2014-04-25 06:24:56 UTC
This is a little bit different error than the one on Windows even though they are very similar.

You can see my results in http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap5-jon-plugin-solaris-intel/SOLARIS_PREFERRED_ARCH=64,jdk=java16_default,label=%28solaris10%26%26x86%29||ews-solaris10-x86/

There was exception that suddenly it succeeded, but I have run the test many times and that was a very rare exception.

Comment 4 Thomas Segismont 2014-04-25 16:36:38 UTC
Solaris 10 is not supported with JON 3.2. See https://access.redhat.com/site/articles/112523

I will install a Solaris 11 VM on my box and try to reproduce.

Comment 6 Thomas Segismont 2014-04-28 14:05:48 UTC
I just finished a test of JON3.2 + EAP5.3 on Solaris 11 Intel. Stop and Start operations work.

Like for bug 1087504, can you please build an environment where I can reproduce this bug? The only environment where this problem shows up is a Jenkins slave and the inventory deleted after the suite is executed

Comment 7 Radim Hatlapatka 2014-04-30 10:00:59 UTC
I have narrowed down that this issue only occurs on Solaris 10 JDK 1.6 64bit JVM (=> -d64 in JAVA_OPTS).

The used java version:
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

Comment 8 Radim Hatlapatka 2014-04-30 13:45:35 UTC
Comment 7 is true for Solaris 10 on Intel architecture

In case of Solaris 10 sparc, it fails on both JDK 1.6 and 1.7

I have even tried Solaris 11 sparc and it failed with a little bit different message:

java.lang.Exception: The server failed to start: ========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /mnt/hudson_workspace/workspace/eap5-jon-plugin-solaris-sparc/81f3f08f/eap5unsecured/jboss-eap-5.3/jboss-as JAVA: java JAVA_OPTS: -Dprogram.name=run.sh -server -d64 -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.net.preferIPv4Stack=true CLASSPATH: /mnt/hudson_workspace/workspace/eap5-jon-plugin-solaris-sparc/81f3f08f/eap5unsecured/jboss-eap-5.3/jboss-as/bin/run.jar ========================================================================= Failed to boot JBoss: java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at org.jboss.Main.processCommandLine(Main.java:280) at org.jboss.Main.boot(Main.java:129) at org.jboss.Main$1.call(Main.java:561) at org.jboss.Main$1.call(Main.java:557) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.lang.Thread.run(Thread.java:595) at org.rhq.core.pc.operation.OperationInvocation.run(OperationInvocation.java:278) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) expected [true] but found [false]

Comment 11 Thomas Segismont 2014-09-02 12:50:41 UTC
(In reply to Radim Hatlapatka from comment #8)
> Failed to boot JBoss: java.lang.UnsupportedClassVersionError: Bad version
> number in .class file at java.lang.ClassLoader.defineClass1(Native Method)

UnsupportedClassVersionError means the Java bytecode version is not supported by this JVM

> /mnt/hudson_workspace/workspace/eap5-jon-plugin-solaris-sparc/81f3f08f/
> eap5unsecured/jboss-eap-5.3/jboss-as JAVA: java JAVA_OPTS:

So it looks like JON was not able to determine JAVA_HOME and uses the default JVM ("JAVA: java").

The only thing I can imagine is that, for some reason, the process environment cannot be determined. This could be because the RHQ Agent is running as a user without the required privilege.

Which user are your running on:
* the RHQ agent?
* the EAP 5.3 instance?

Can you attach the agent log at DEBUG level?

Comment 15 Larry O'Leary 2015-01-09 23:52:27 UTC
Closing as WORKSFORME as this issue can not be reproduced on a supported JBoss EAP 5 version.