Bug 906041

Summary: [jboss-as-7] Response time metrics can not be collected due to java.lang.UnsupportedOperationException
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: Plugin -- JBoss EAP 6Assignee: Larry O'Leary <loleary>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: urgent    
Version: JON 3.1.2CC: dowoods, lzoubek
Target Milestone: ER01   
Target Release: JON 3.2.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 906418 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 864213    
Bug Blocks: 906418    

Description Larry O'Leary 2013-01-30 16:46:42 UTC
Description of problem:
After enabling response time metrics for a web application deployed to EAP 6 no calltime data is made available under the WARs calltime monitoring page.

The response time log file contains valid data and the response time module appears to be functioning in the EAP 6 server. However, no data makes it to the RHQ server.

The agent log shows that the collection of calltime data fails with the following stack:

    WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=10122, uuid=df343d2f-6280-4ee3-ac73-468a100d7898, type={JBossAS7}Web Runtime, key=deployment=jboss-as-helloworld.war,subsystem=web, name=web, parent=jboss-as-helloworld.war], requests=[ScheduledMeasurementInfo[res=10122, name=responseTime, sched=10694]], report.size()=0
    java.lang.UnsupportedOperationException
        at java.util.Collections$UnmodifiableCollection$1.remove(Collections.java:1028)
        at org.rhq.modules.plugins.jbossas7.WebRuntimeComponent.getValues(WebRuntimeComponent.java:84)
        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)


Version-Release number of selected component (if applicable):
4.4.0.JON312GA

How reproducible:
Always

Steps to Reproduce:
1.  Install `rhq-rtfilter-module.zip` into EAP `modules` directory:

        cd "${JBOSS_HOME}/modules"
        unzip "${TESTENV_DIR}/jon-server/jbossas/server/default/deploy/rhq.ear/rhq-downloads/connectors/rhq-rtfilter-module.zip"

2.  Add the `org.rhq.helpers.rhq-rtfilter` module to the `global-modules` section of the `urn:jboss:domain:ee:1.1` subsystem in `standalone.xml`:

        <profile...					
	        <subsystem xmlns="urn:jboss:domain:ee:1.1">	
		        <global-modules>
		            <module name="org.rhq.helpers.rhq-rtfilter" slot="main"/>
		        </global-modules>
	        </subsystem>
        </profile>

    The subsystem defintion should already exist and you should only need to add the global-modules definition.
    
3.  Install `rhq-rtfilter-subsystem-module.zip` into EAP `modules` directory:

        cd "${JBOSS_HOME}/modules"
        unzip "${TESTENV_DIR}/jon-server/jbossas/server/default/deploy/rhq.ear/rhq-downloads/connectors/rhq-rtfilter-subsystem-module.zip"

4.  Add the `org.rhq.helpers.rhq-rtfilter-subsystem` module to the  `extensions` element of standalone.xml:

        <extension module="org.rhq.helpers.rhq-rtfilter-subsystem"/>

5.  Add the `urn:rhq:rtfilter:1.0` subsystem to the `profile` element of standalone.xml:

        <subsystem xmlns="urn:rhq:rtfilter:1.0">
            <!-- Optional parameters. 
              
               <init-param>
                   <param-name>chopQueryString</param-name>
                   <param-value>true</param-value>
               </init-param>
               <init-param>
                   <param-name>logDirectory</param-name>
                  <param-value>/tmp</param-value>
               </init-param>
               <init-param>
                   <param-name>logFilePrefix</param-name>
                   <param-value>localhost_7080_</param-value>
               </init-param>
               <init-param>
                   <param-name>dontLogRegEx</param-name>
                   <param-value></param-value>
               </init-param>
               <init-param>
                  <param-name>matchOnUriOnly</param-name>
                  <param-value>true</param-value>
               </init-param>
               <init-param>
                   <param-name>timeBetweenFlushesInSec</param-name>
                   <param-value>73</param-value>
               </init-param>
               <init-param>
                   <param-name>flushAfterLines</param-name>
                   <param-value>13</param-value>
               </init-param>
               <init-param>
                   <param-name>maxLogFileSize</param-name>
                   <param-value>5242880</param-value>
               </init-param>       
        -->
        </subsystem>

6.  Start EAP 6 standalone server.
7.  Build and deploy `jboss-as-helloworld.war` from Quickstarts.

    Any WAR will do as long as it has been deployed and enabled in the EAP 6 server.
    
8.  Start JBoss ON system.
9.  Import EAP 6 standalone server into inventory.
10. Enable the response time metrics for the jboss-as-helloworld.war's web resource.
    
    1.  Navigate to the EAP 6 standalone resource's *Deployments / jboss-as-helloworld.war / web* resource's monitoring schedules page.
    2.  Set the *response time* *calltime* metric's collection interval to 1 minute.
    3.  Navigate to the EAP 6 standalone resource's *Deployments / jboss-as-helloworld.war / web* resource's inventory connection settings page.
    4.  Set *response time log file* to the complete path of the response time log file for the jboss-as-helloworld.war.
    
        This should be in `<JBOSS_HOME>/standalone/log/rt/jboss-as-helloworld_rt.log`.
11. Wait for a couple of minutes to be certain that the collection interval has elapsed.

  
Actual results:
No response time data is collected and the agent log contains the following warning each time the collection interval elapses (1 minute):
    WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=10122, uuid=df343d2f-6280-4ee3-ac73-468a100d7898, type={JBossAS7}Web Runtime, key=deployment=jboss-as-helloworld.war,subsystem=web, name=web, parent=jboss-as-helloworld.war], requests=[ScheduledMeasurementInfo[res=10122, name=responseTime, sched=10694]], report.size()=0
    java.lang.UnsupportedOperationException
        at java.util.Collections$UnmodifiableCollection$1.remove(Collections.java:1028)
        at org.rhq.modules.plugins.jbossas7.WebRuntimeComponent.getValues(WebRuntimeComponent.java:84)
        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)

Expected results:
Response time data should appear on the jboss-as-helloworld.war's web resource's monitoring calltime page.

Additional info:
The root cause of the issue is that we are attempting to modify a UnmodifiableCollection in org.rhq.modules.plugins.jbossas7.WebRuntimeComponent.getValues.

Comment 1 Lukas Krejci 2013-01-31 15:36:11 UTC
This should be fixed in master by BZ 864213 and BZ 864619. The latter BZ seems to have been applied to the JON branch, unlike the former one.

Cherry-picking commit b44ad1b934bbc80a52bfa1d4202294a63a94802c over the vanilla JON312GA codebase fixed the issue for me and I was able to collect the calltime metrics.

Comment 2 Larry O'Leary 2013-02-04 21:09:29 UTC
This fix will be picked up from master when we branch the 3.2 release from master. Moving to MODIFIED as this should be in 3.2.0.ER1.

Comment 3 Larry O'Leary 2013-02-06 15:49:15 UTC
Appears the upstream fix introduces a new failure:

2013-02-06 09:36:54,077 WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [JBossAS7 Standalone Server] Resources - failed after 150 ms.
java.lang.Exception: Discovery component invocation failed.
	at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:297)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoSuchMethodError: org.rhq.modules.plugins.jbossas7.AS7CommandLine.isArgumentsParsed()Z
	at org.rhq.modules.plugins.jbossas7.AS7CommandLine.getAppServerArguments(AS7CommandLine.java:89)
	at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.setStartScriptPluginConfigProps(BaseProcessDiscovery.java:277)
	at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.buildResourceDetails(BaseProcessDiscovery.java:198)
	at org.rhq.modules.plugins.jbossas7.StandaloneASDiscovery.buildResourceDetails(StandaloneASDiscovery.java:84)
	at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.discoverResources(BaseProcessDiscovery.java:136)
	at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:293)
	... 5 more


Setting to ON_DEV and will update the upstream bug(s).

Comment 4 Larry O'Leary 2013-02-06 15:50:43 UTC
Ooops, wrong BZ. Comment 3 was intended for bug 907577. Setting back to MODIFIED.

Comment 5 Libor Zoubek 2013-11-01 13:41:33 UTC
verified on JON 3.2.ER4