Bug 876656 - AS servers can not be discovered when started as a service (no start script)
AS servers can not be discovered when started as a service (no start script)
Status: CLOSED CURRENTRELEASE
Product: JBoss Operations Network
Classification: JBoss
Component: Agent (Show other bugs)
JON 3.1.0
All Linux
high Severity high
: ---
: JON 3.1.2
Assigned To: Thomas Segismont
Mike Foley
:
Depends On: 876658
Blocks: 889628
  Show dependency treegraph
 
Reported: 2012-11-14 11:45 EST by Larry O'Leary
Modified: 2013-09-11 07:04 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 876658 (view as bug list)
Environment:
Last Closed: 2013-09-11 07:04:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Example System V init script to demonstrate this bug (1.71 KB, application/octet-stream)
2012-11-26 19:12 EST, Larry O'Leary
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 162643 None None None 2012-11-14 11:45:43 EST

  None (edit)
Description Larry O'Leary 2012-11-14 11:45:18 EST
Description of problem:
AS server discovery fails with the following exception:

WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [JBossAS Server] Resources - failed after 20 ms.
java.lang.Exception: Discovery component invocation failed.
        at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:297)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
        at java.lang.String.charAt(String.java:686)
        at org.rhq.core.pluginapi.util.ServerStartScriptDiscoveryUtility.getStartScriptIndex(ServerStartScriptDiscoveryUtility.java:295)
        at org.rhq.core.pluginapi.util.ServerStartScriptDiscoveryUtility.getStartScript(ServerStartScriptDiscoveryUtility.java:86)
        at org.rhq.plugins.jbossas5.ApplicationServerDiscoveryComponent.setStartScriptPluginConfigProps(ApplicationServerDiscoveryComponent.java:420)
        at org.rhq.plugins.jbossas5.ApplicationServerDiscoveryComponent.createResourceDetails(ApplicationServerDiscoveryComponent.java:407)
        at org.rhq.plugins.jbossas5.ApplicationServerDiscoveryComponent.discoverExternalJBossAsProcesses(ApplicationServerDiscoveryComponent.java:340)
        at org.rhq.plugins.jbossas5.ApplicationServerDiscoveryComponent.discoverResources(ApplicationServerDiscoveryComponent.java:170)
        at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:293)
        ... 5 more


Version-Release number of selected component (if applicable):
4.4.0.JON310GA

How reproducible:
Always

Steps to Reproduce:
Not sure yet but the key is that the parent process of the AS JVM process needs to be something similar to the Linux init service.

  
Actual results:
Discovery of the AS server fails and the agent log contains a warning with the following exception: java.lang.StringIndexOutOfBoundsException: String index out of range: 0

Expected results:
Discovery of the AS server should be successful and the AS start script should be unset.

Additional info:
This issue is a direct result of the parent process that started the JVM not being the expected run.sh or standalone.sh/domain.sh script. For example, if the user starts the JVM directly from the command-line without using a shell script; or as boot when the init system starts its configured services.

In the case of the failure, the parent process command line is:

init [3]                                   


Whereas, what we expect to see is:

/bin/sh ./run.sh -c production -b 0.0.0.0


Simply put, ServerStartScriptDiscoveryUtility.getStartScriptIndex should not assume that the second argument (or any argument for that matter) has a length of 1 or more.

-            if (serverParentProcessCommandLine[startScriptIndex].charAt(0) != OPTION_PREFIX) {
+            if (serverParentProcessCommandLine[startScriptIndex] != null
+                && serverParentProcessCommandLine[startScriptIndex].length() > 0
+                && serverParentProcessCommandLine[startScriptIndex].charAt(0) != OPTION_PREFIX) {
Comment 2 Larry O'Leary 2012-11-26 19:10:44 EST
There are many options available to reproduce this failure. I used EAP 5.1.2 however the same process can be used for EAP 6. The platform used is RHEL 5.7 running in headless mode (text mode/runlevel 3). 

One option is to simply modify some environment variable and have 'root' execute them:

# Variables that may need to change
JAVA_HOME="/usr/lib/jvm/java"
JBOSS_HOME="/opt/jboss/eap/jboss-eap-5.1/jboss-as"
RUNAS="jonadmin"
JBOSSCONF="default"
BIND_ADDR="0.0.0.0"
JNDI_PORT=1099

# Variables that probably won't change
JAVA="${JAVA_HOME}/bin/java"
JAVA_OPTS="-Dprogram.name=run.sh -server -Xms128m -Xmx768m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.net.preferIPv4Stack=true"
JBOSS_ENDORSED_DIRS="${JBOSS_HOME}/lib/endorsed"
JBOSS_CLASSPATH="${JBOSS_HOME}/bin/run.jar:${JAVA_HOME}/lib/tools.jar"

    eval su - $RUNAS -c \'\"$JAVA\" $JAVA_OPTS -Djava.endorsed.dirs=\"$JBOSS_ENDORSED_DIRS\" -classpath \"$JBOSS_CLASSPATH\" org.jboss.Main -c ${JBOSSCONF} -b ${BIND_ADDR} "&"\'



Another option is to use a SysV style script. The attached example SysV script can be copied into /etc/init.d and then run:

chkconfig --add jboss-init-run-default

Then update /etc/init.d/jboss-init-run-default script with necessary parameter values and then:

service jboss-init-run-default start


In both cases the process output will look like:

jonadmin 23059     1 17 22:55 ?        00:00:41 /usr/lib/jvm/java/bin/java -Dprogram.name=run.sh -server -Xms128m -Xmx768m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss/eap/jboss-eap-5.1/jboss-as/lib/endorsed -classpath /opt/jboss/eap/jboss-eap-5.1/jboss-as/bin/run.jar:/usr/lib/jvm/java/lib/tools.jar org.jboss.Main -c default -b 0.0.0.0

Pay special attention to the parent process ID (the 3rd column). It should be 1. The parent process should look like:

root         1     0  0 Oct11 ?        00:00:01 init [3]         



Once EAP is running, the agent will fail to discover it with the exception mentioned in the description.
Comment 3 Larry O'Leary 2012-11-26 19:12:02 EST
Created attachment 652410 [details]
Example System V init script to demonstrate this bug
Comment 4 Jay Shaughnessy 2012-12-06 14:42:10 EST
On behalf of tsegismont:

commit 5c94096f961423286a72a6bf7aeb74aff1485b75
Author: Jay Shaughnessy <jshaughn@jshaughn.csb>
Date:   Thu Dec 6 14:39:46 2012 -0500

    Add some null/empty string protection, even though the string really should
    never be empty.
    
    Cherry pick of master commit edb9cf7
Comment 5 Simeon Pinder 2012-12-10 07:40:47 EST
Moving to ON_QA as available for test in 3.1.2.ER4 or greater: https://brewweb.devel.redhat.com//buildinfo?buildID=246861
Comment 6 Sunil Kondkar 2013-01-03 06:13:38 EST
Verified on JON 3.1.2 ER6.

Used EAP 5.1.2 and attached example SysV script. Platform used is RHEL 5.7 running in runlevel 3.
Started EAP with 'service jboss-init-run-default start'. Started agent and verified that EAP is successfully discovered and imported without any exception in agent log.

Also verified that EAP Resource->Inventory->Connection Settings shows 'Start Script' as unset.

Note You need to log in before you can comment on or make changes to this bug.