Bug 906418

Summary: [jboss-as-7] Response time metrics can not be collected due to java.lang.UnsupportedOperationException
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: Plugin -- JBoss EAP 6Assignee: Larry O'Leary <loleary>
Status: CLOSED NEXTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: urgent    
Version: JON 3.1.2CC: dowoods
Target Milestone: ER01   
Target Release: JON 3.1.3   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 906041
: 906422 (view as bug list) Environment:
Last Closed: 2013-09-06 02:24:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 906041    
Bug Blocks: 906422    

Description Larry O'Leary 2013-01-31 15:44:36 UTC
+++ This bug was initially created as a clone of Bug #906041 +++

Description of problem:
After enabling response time metrics for a web application deployed to EAP 6 no calltime data is made available under the WARs calltime monitoring page.

The response time log file contains valid data and the response time module appears to be functioning in the EAP 6 server. However, no data makes it to the RHQ server.

The agent log shows that the collection of calltime data fails with the following stack:

    WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=10122, uuid=df343d2f-6280-4ee3-ac73-468a100d7898, type={JBossAS7}Web Runtime, key=deployment=jboss-as-helloworld.war,subsystem=web, name=web, parent=jboss-as-helloworld.war], requests=[ScheduledMeasurementInfo[res=10122, name=responseTime, sched=10694]], report.size()=0
    java.lang.UnsupportedOperationException
        at java.util.Collections$UnmodifiableCollection$1.remove(Collections.java:1028)
        at org.rhq.modules.plugins.jbossas7.WebRuntimeComponent.getValues(WebRuntimeComponent.java:84)
        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)


Version-Release number of selected component (if applicable):
4.4.0.JON312GA

How reproducible:
Always

Steps to Reproduce:
1.  Install `rhq-rtfilter-module.zip` into EAP `modules` directory:

        cd "${JBOSS_HOME}/modules"
        unzip "${TESTENV_DIR}/jon-server/jbossas/server/default/deploy/rhq.ear/rhq-downloads/connectors/rhq-rtfilter-module.zip"

2.  Add the `org.rhq.helpers.rhq-rtfilter` module to the `global-modules` section of the `urn:jboss:domain:ee:1.1` subsystem in `standalone.xml`:

        <profile...					
	        <subsystem xmlns="urn:jboss:domain:ee:1.1">	
		        <global-modules>
		            <module name="org.rhq.helpers.rhq-rtfilter" slot="main"/>
		        </global-modules>
	        </subsystem>
        </profile>

    The subsystem defintion should already exist and you should only need to add the global-modules definition.
    
3.  Install `rhq-rtfilter-subsystem-module.zip` into EAP `modules` directory:

        cd "${JBOSS_HOME}/modules"
        unzip "${TESTENV_DIR}/jon-server/jbossas/server/default/deploy/rhq.ear/rhq-downloads/connectors/rhq-rtfilter-subsystem-module.zip"

4.  Add the `org.rhq.helpers.rhq-rtfilter-subsystem` module to the  `extensions` element of standalone.xml:

        <extension module="org.rhq.helpers.rhq-rtfilter-subsystem"/>

5.  Add the `urn:rhq:rtfilter:1.0` subsystem to the `profile` element of standalone.xml:

        <subsystem xmlns="urn:rhq:rtfilter:1.0">
            <!-- Optional parameters. 
              
               <init-param>
                   <param-name>chopQueryString</param-name>
                   <param-value>true</param-value>
               </init-param>
               <init-param>
                   <param-name>logDirectory</param-name>
                  <param-value>/tmp</param-value>
               </init-param>
               <init-param>
                   <param-name>logFilePrefix</param-name>
                   <param-value>localhost_7080_</param-value>
               </init-param>
               <init-param>
                   <param-name>dontLogRegEx</param-name>
                   <param-value></param-value>
               </init-param>
               <init-param>
                  <param-name>matchOnUriOnly</param-name>
                  <param-value>true</param-value>
               </init-param>
               <init-param>
                   <param-name>timeBetweenFlushesInSec</param-name>
                   <param-value>73</param-value>
               </init-param>
               <init-param>
                   <param-name>flushAfterLines</param-name>
                   <param-value>13</param-value>
               </init-param>
               <init-param>
                   <param-name>maxLogFileSize</param-name>
                   <param-value>5242880</param-value>
               </init-param>       
        -->
        </subsystem>

6.  Start EAP 6 standalone server.
7.  Build and deploy `jboss-as-helloworld.war` from Quickstarts.

    Any WAR will do as long as it has been deployed and enabled in the EAP 6 server.
    
8.  Start JBoss ON system.
9.  Import EAP 6 standalone server into inventory.
10. Enable the response time metrics for the jboss-as-helloworld.war's web resource.
    
    1.  Navigate to the EAP 6 standalone resource's *Deployments / jboss-as-helloworld.war / web* resource's monitoring schedules page.
    2.  Set the *response time* *calltime* metric's collection interval to 1 minute.
    3.  Navigate to the EAP 6 standalone resource's *Deployments / jboss-as-helloworld.war / web* resource's inventory connection settings page.
    4.  Set *response time log file* to the complete path of the response time log file for the jboss-as-helloworld.war.
    
        This should be in `<JBOSS_HOME>/standalone/log/rt/jboss-as-helloworld_rt.log`.
11. Wait for a couple of minutes to be certain that the collection interval has elapsed.

  
Actual results:
No response time data is collected and the agent log contains the following warning each time the collection interval elapses (1 minute):
    WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=10122, uuid=df343d2f-6280-4ee3-ac73-468a100d7898, type={JBossAS7}Web Runtime, key=deployment=jboss-as-helloworld.war,subsystem=web, name=web, parent=jboss-as-helloworld.war], requests=[ScheduledMeasurementInfo[res=10122, name=responseTime, sched=10694]], report.size()=0
    java.lang.UnsupportedOperationException
        at java.util.Collections$UnmodifiableCollection$1.remove(Collections.java:1028)
        at org.rhq.modules.plugins.jbossas7.WebRuntimeComponent.getValues(WebRuntimeComponent.java:84)
        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)

Expected results:
Response time data should appear on the jboss-as-helloworld.war's web resource's monitoring calltime page.

Additional info:
The root cause of the issue is that we are attempting to modify a UnmodifiableCollection in org.rhq.modules.plugins.jbossas7.WebRuntimeComponent.getValues.

--- Additional comment from Lukas Krejci on 2013-01-31 10:36:11 EST ---

This should be fixed in master by BZ 864213 and BZ 864619. The latter BZ seems to have been applied to the JON branch, unlike the former one.

Cherry-picking commit b44ad1b934bbc80a52bfa1d4202294a63a94802c over the vanilla JON312GA codebase fixed the issue for me and I was able to collect the calltime metrics.

Comment 1 Larry O'Leary 2013-02-04 22:08:34 UTC
Committed to release/jon3.1.x branch as http://git.fedorahosted.org/cgit/rhq/rhq.git/commit/?id=37bd3a21cfc97e7fe984da6fcddfe78abcd9be19

commit 37bd3a21cfc97e7fe984da6fcddfe78abcd9be19
Author: John Mazzitelli <mazz>
Date:   Mon Oct 8 17:48:27 2012 -0400

    [BZ 864213] fix the web resource's ability to collect response time data.
    (cherry picked from master commit b44ad1b934bbc80a52bfa1d4202294a63a94802c)



Setting status as MODIFIED as this will be in the first ER release for 3.1.3 if there ever is one.

Comment 2 Larry O'Leary 2013-02-06 15:48:15 UTC
Appears the upstream fix introduces a new failure:

2013-02-06 09:36:54,077 WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [JBossAS7 Standalone Server] Resources - failed after 150 ms.
java.lang.Exception: Discovery component invocation failed.
	at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:297)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NoSuchMethodError: org.rhq.modules.plugins.jbossas7.AS7CommandLine.isArgumentsParsed()Z
	at org.rhq.modules.plugins.jbossas7.AS7CommandLine.getAppServerArguments(AS7CommandLine.java:89)
	at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.setStartScriptPluginConfigProps(BaseProcessDiscovery.java:277)
	at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.buildResourceDetails(BaseProcessDiscovery.java:198)
	at org.rhq.modules.plugins.jbossas7.StandaloneASDiscovery.buildResourceDetails(StandaloneASDiscovery.java:84)
	at org.rhq.modules.plugins.jbossas7.BaseProcessDiscovery.discoverResources(BaseProcessDiscovery.java:136)
	at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:293)
	... 5 more


Setting to ON_DEV and will update the upstream bug(s).

Comment 3 Larry O'Leary 2013-02-06 15:50:57 UTC
Ooops, wrong BZ. Comment 2 was intended for bug 907577. Setting back to MODIFIED.

Comment 4 Larry O'Leary 2013-09-06 02:24:51 UTC
Closing as there will not be a 3.1.3 release. This is being tracked for 3.2 in the 'depends on' field.