Bug 889628

Summary: Discovery of EAP fails when EAP is running as a Windows service
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: Plugin -- JBoss EAP 5, Plugin -- JBoss EAP 6Assignee: Thomas Segismont <tsegismo>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: high    
Version: JON 3.1.1CC: skondkar
Target Milestone: ---Keywords: NeedsTestCase
Target Release: JON 3.1.2   
Hardware: All   
OS: Windows   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-11 11:01:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 876656    
Bug Blocks:    

Description Larry O'Leary 2012-12-22 00:16:46 UTC
Description of problem:
If AS/EAP server has been started as a service, the agent fails to discover it and spits out the following error:

    2012-12-21 04:29:24,187 ERROR [ResourceDiscoveryComponent.invoker.daemon-1] (rhq.modules.plugins.jbossas7.HostControllerDiscovery)- Discovery of a JBossAS7 Host Controller Resource failed for process: pid=[1904], name=[java], ppid=[2496].
    java.lang.ArrayIndexOutOfBoundsException: 2
	    at org.rhq.core.pluginapi.util.ServerStartScriptDiscoveryUtility.getStartScriptIndex(ServerStartScriptDiscoveryUtility.java:299)
	    at org.rhq.core.pluginapi.util.ServerStartScriptDiscoveryUtility.getStartScript(ServerStartScriptDiscoveryUtility.java:86)
	    at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.setStartScriptPluginConfigProps(BaseProcessDiscovery.java:251)
	    at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.buildResourceDetails(BaseProcessDiscovery.java:197)
	    at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.discoverResources(BaseProcessDiscovery.java:136)
	    at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
	    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	    at java.lang.reflect.Method.invoke(Unknown Source)
	    at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:293)
	    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	    at java.util.concurrent.FutureTask.run(Unknown Source)
	    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	    at java.lang.Thread.run(Unknown Source)


Version-Release number of selected component (if applicable):
4.4.0.JON311GA

How reproducible:
Always

Steps to Reproduce:
1.  Install EAP 6
2.  Either setup EAP 6 domain to run as a Windows service or simulate it by executing the command it will use:
    
    Simulated:

        cd C:\jboss\eap\jboss-eap-6.0\bin
        cmd "/c domain.bat -b 0.0.0.0"
        
    Run as a service:        

        cd C:\jboss\eap\jboss-eap-6.0\modules\native\sbin
        prunsrv install "JBossEAP6Domain" --DisplayName="JBoss EAP 6 Domain" --Description "JBoss Enterprise Application Platform (EAP) 6 - Domain Controller" --LogLevel=INFO --LogPath=C:\jboss\eap\jboss-eap-6.0\domain\log --LogPrefix=service --StdOutput=auto --StdError=auto --StartMode=exe --StartImage=cmd.exe --StartPath=C:\jboss\eap\jboss-eap-6.0\bin ++StartParams="/c domain.bat -b 0.0.0.0" --StopMode=exe --StopImage=cmd.exe --StopPath=C:\jboss\eap\jboss-eap-6.0\bin  ++StopParams="/c jboss-cli.bat --connect command=/host=master:shutdown"

3.  If using a Windows service for EAP, start the server:

        net start JBossEAP6Domain
        
4.  Start ON agent
  
Actual results:
EAP 6 host controller is not discovered and the agent log contains the following error:
    2012-12-21 04:29:24,187 ERROR [ResourceDiscoveryComponent.invoker.daemon-1] (rhq.modules.plugins.jbossas7.HostControllerDiscovery)- Discovery of a JBossAS7 Host Controller Resource failed for process: pid=[1904], name=[java], ppid=[2496].
    java.lang.ArrayIndexOutOfBoundsException: 2


Expected results:
EAP 6 host controller should be discovered and apper in the discovery queue.


Additional info:
Using `wmic /OUTPUT:C:\ps.out PROCESS get Caption,Commandline,Processid` we can see the following relevant process info:

    Caption	CommandLine	ProcessId
    java.exe	java "-D[Host Controller]" -Dorg.jboss.boot.log.file=C:\jboss\eap\jboss-eap-6.0\domain\log\host-controller.log -Dlogging.configuration=file:C:\jboss\eap\jboss-eap-6.0\domain/configuration/logging.properties -Xms64M -Xmx512M -XX:MaxPermSize=256M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -server -jar C:\jboss\eap\jboss-eap-6.0\jboss-modules.jar -mp C:\jboss\eap\jboss-eap-6.0\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp C:\jboss\eap\jboss-eap-6.0\modules --pc-address 127.0.0.1 --pc-port 49310 -default-jvm java -b 0.0.0.0 -Djboss.home.dir=C:\jboss\eap\jboss-eap-6.0	1904
    java.exe	"java"  -Xms64M -Xmx512M -XX:MaxPermSize=256M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -server  "-Dorg.jboss.boot.log.file=C:\jboss\eap\jboss-eap-6.0\domain\log\process-controller.log"  "-Dlogging.configuration=file:C:\jboss\eap\jboss-eap-6.0\domain/configuration/logging.properties"     -jar "C:\jboss\eap\jboss-eap-6.0\jboss-modules.jar"     -mp "C:\jboss\eap\jboss-eap-6.0\modules"      org.jboss.as.process-controller     -jboss-home "C:\jboss\eap\jboss-eap-6.0"     -jvm "java"     -mp "C:\jboss\eap\jboss-eap-6.0\modules"     --     "-Dorg.jboss.boot.log.file=C:\jboss\eap\jboss-eap-6.0\domain\log\host-controller.log"     "-Dlogging.configuration=file:C:\jboss\eap\jboss-eap-6.0\domain/configuration/logging.properties"     -Xms64M -Xmx512M -XX:MaxPermSize=256M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -server     --     -default-jvm "java"     -b 0.0.0.0	2496
    cmd.exe	cmd.exe "/c domain.bat -b 0.0.0.0"	1652


Using Byteman, we can see what the command-line looks like that causes the parsing failure:
    serverParentProcessCommandLine[] = [cmd.exe, /c domain.bat -b 0.0.0.0]


What this boils down to is that we have two parameters in the serverParentProcessCommandLine array. We always skip the first... and then we skip try to find the first remaining parameter that does not start with a '/' or '-'. In this case, that means we skip all the parameters. We then attempt to use the non-existent 3rd parameter as the startScriptIndex and then getStartScriptIndex fails due to the index being out of bounds.

This is actually very similar to the issue that was identified in bug 876656 and the fix in bug 876656 actually fixes this issue too. Specifically due to the following change:

        // for whatever unanticipated reason, we advanced past all of the args
        if (startScriptIndex == serverParentProcessCommandLine.length) {
            return null;
        }

Comment 2 Larry O'Leary 2012-12-22 00:20:32 UTC
Assigning to Thomas considering he did the back-port for the dependent bug that fixes this issue too. Although this issue is different the fix just so happened to also be part of the fix done in bug 876656.

Putting ON_QA as this fix is already in 3.1.2.ER4.

Comment 3 Sunil Kondkar 2012-12-31 17:42:43 UTC
Verified on JON 3.1.2 ER6

Installed jboss-eap-native-utils for windows and started EAP6 domain as windows service. Followed the steps and verified that EAP 6 host controller is discovered and appear in the discovery queue without error in agent log.