Bug 1011224

Summary: ProxyFactory.getResource does not always return a valid resource proxy object
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: CLIAssignee: Lukas Krejci <lkrejci>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: JON 3.1.2CC: fbrychta, lkrejci
Target Milestone: ER01   
Target Release: JON 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1012156 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 829944    
Bug Blocks: 1012156    

Description Larry O'Leary 2013-09-23 20:19:04 UTC
Description of problem:
When invoking a CLI script that uses ProxyFactory.getResource(resourceId) the returned object doesn't always reflect the expected resource. This will result in unexpected exceptions when invoking methods specific to a resource type:

Exception in thread "main" org.rhq.enterprise.client.script.CLIScriptException: javax.script.ScriptException: sun.org.mozilla.javascript.EcmaError: TypeError: Cannot read property "displayValue" from undefined (<Unknown source>#12) in <Unknown source> at line number 12
	at org.rhq.enterprise.client.commands.ScriptCommand.executeScriptFile(ScriptCommand.java:298)
	at org.rhq.enterprise.client.commands.ScriptCommand.execute(ScriptCommand.java:108)
	at org.rhq.enterprise.client.ClientMain$StartupConfiguration.process(ClientMain.java:141)
	at org.rhq.enterprise.client.ClientMain.main(ClientMain.java:161)
Caused by: javax.script.ScriptException: sun.org.mozilla.javascript.EcmaError: TypeError: Cannot read property "displayValue" from undefined (<Unknown source>#12) in <Unknown source> at line number 12
	at com.sun.script.javascript.RhinoScriptEngine.eval(RhinoScriptEngine.java:179)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:249)
	at org.rhq.enterprise.client.commands.ScriptCommand.executeScriptFile(ScriptCommand.java:287)
	... 3 more
Caused by: sun.org.mozilla.javascript.EcmaError: TypeError: Cannot read property "displayValue" from undefined (<Unknown source>#12)
	at sun.org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3654)
	at sun.org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3632)
	at sun.org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3660)
	at sun.org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3679)
	at sun.org.mozilla.javascript.ScriptRuntime.undefReadError(ScriptRuntime.java:3692)
	at sun.org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1421)
	at sun.org.mozilla.javascript.gen.c197._c0(<Unknown source>:12)
	at sun.org.mozilla.javascript.gen.c197.call(<Unknown source>)
	at sun.org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:398)
	at com.sun.script.javascript.RhinoScriptEngine$1.superDoTopCall(RhinoScriptEngine.java:112)
	at com.sun.script.javascript.RhinoScriptEngine$1.doTopCall(RhinoScriptEngine.java:105)
	at sun.org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3065)
	at sun.org.mozilla.javascript.gen.c197.call(<Unknown source>)
	at sun.org.mozilla.javascript.gen.c197.exec(<Unknown source>)
	at sun.org.mozilla.javascript.Context.evaluateReader(Context.java:1135)
	at com.sun.script.javascript.RhinoScriptEngine.eval(RhinoScriptEngine.java:169)
	... 5 more


Version-Release number of selected component (if applicable):
4.4.0.JON312GA

How reproducible:
Every few invocations (frequently enough)

Steps to Reproduce:
1.  Install and start EAP 6 standalone-full.xml instance.
2.  Add test JMS Queues to EAP server:

        "${JBOSS_HOME}/bin/jboss-cli.sh" --connect --command="jms-queue add --queue-address=queue-01 --entries=queue/queue-01"

3.  Start JBoss ON system.
4.  Import EAP 6 server into inventory.
5.  Configure connection settings for EAP resource.
6.  Create test CLI script:

        cat >/tmp/printQueueMetrics.js <<EOF
var run = 0;
for (run=1; run <= 1500; run++) {
    var date = java.util.Date();
    println("Run " + run + " at " + date + "...");
    var criteria = new ResourceCriteria();
    criteria.addFilterResourceTypeName("JMS Queue");
    criteria.setStrict(true);
    criteria.fetchSchedules(true);
    var resources = ResourceManager.findResourcesByCriteria(criteria);

    if (resources.size() > 0) {
        var resource = resources.get(0);
        var queue = ProxyFactory.getResource(resource.getId());
        println("                           queue = " + queue);
        println("             queue.messagesAdded = " + queue.messagesAdded);
        var schedules = resource.getSchedules();
        var metricDefIds = [];
        for (var iterator = schedules.iterator(); iterator.hasNext();) {
            var schedule = iterator.next();
            metricDefIds.push(schedule.getDefinition().getId());
        }
        metrics = MeasurementDataManager.findLiveData(resource.getId(), metricDefIds);
        for (var iterator = metrics.iterator(); iterator.hasNext();) {
            var metric = iterator.next();
            if ( "messages-added".equals(metric.getName()) ) {
                println("MeasurementDataManager -> metric.getValue() = " + metric.getValue());
            }
        }
        try {
            println("queue.messagesAdded.displayValue = " + queue.messagesAdded.displayValue);
        } catch(e) {
            println("queue.messagesAdded.displayValue = Threw Exception " + e);
            println("FAILED ON RUN: " + run);
            break;
        }
    }
}
EOF

7.  Execute CLI script:

        rhq-remoting-cli-4.4.0.JON312GA/bin/rhq-cli.sh -u rhqadmin -p rhqadmin -s localhost -t 7080 -f /tmp/printQueueMetrics.js

Actual results:
Run 13 at Mon Sep 23 15:05:05 CDT 2013...
                           queue = [10097] queue-01 (JMS Queue::JBossAS7)
             queue.messagesAdded = undefined
MeasurementDataManager -> metric.getValue() = 0.0
queue.messagesAdded.displayValue = Threw Exception TypeError: Cannot read property "displayValue" from undefined
FAILED ON RUN: 13

queue.messagesAdded is undefined
queue.messagesAdded.displayValue failed due to: Cannot read property "displayValue" from undefined


Expected results:
Run 13 at Mon Sep 23 15:05:05 CDT 2013...
                           queue = [10097] queue-01 (JMS Queue::JBossAS7)
             queue.messagesAdded = Messages Added
MeasurementDataManager -> metric.getValue() = 0.0
queue.messagesAdded.displayValue = 0.0

queue.messagesAdded shouild be Messages Added
queue.messagesAdded.displayValue should be 0.0

Additional info:
It is not clear what is causing this issue. First off, when executing the above test script using EJB3 Session Bean types, 2000 invocations didn't not reveal any issues. However, when invoking the call on JMS Queue types, it happens 1 out of every 20 or so runs. However, there have been instances where the test could be executed over 530 times before a failure occurred.

Something else to note, it seems that once the failure occurs, this failure affects all subsequent calls to the instance variables. This same issue also impacts the proxy methods too. So, whether you call getMessagesAdded() or simply reference the variable messagesAdded, the result it the same. messagesAdded is simply undefined. 

The extra call to MeasurementDataManager.findLiveData(resource.getId(), metricDefIds) is not needed. It is only there to demonstrate that a value is actually being returned and the resource is still intact. The hope was to find a workaround that could be used but this is not an adequate workaround considering that it requires the agent to be running and available at the time of the call.

Comment 1 Lukas Krejci 2013-09-24 14:30:54 UTC
I've been trying in vain to reproduce this the whole day today.
I tried EAP 6.0 and 6.1, Oracle Java 6 and 7 and OpenJDK 7 but I still didn't see a single instance of this error.

Could you point me to some environment you have set up that exhibits this?

Comment 2 Larry O'Leary 2013-09-24 14:53:09 UTC
Environment I used is:

    java version "1.6.0_20"
    OpenJDK Runtime Environment (IcedTea6 1.9.10) (fedora-55.1.9.10.fc14-x86_64)
    OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

    On Fedora 14 running kernel: 2.6.35.14-106.fc14.x86_64

Comment 3 Lukas Krejci 2013-09-25 16:34:05 UTC
I finally managed to reproduce this.

I could either reproduce it in environments where all CLI, server and agent are co-located on a single box and are running on Java6 (1.6.0_24) (and at the same time that box is NOT my own work machine ;) ) or when CLI is remote and runs on OpenJDK 1.7.0_25 but the server and agent are still on 1.6.0_24.

Not sure why it seems to happen more on Java6 or over remotes.

Additionally, this error message is generated in the rhq-client.log at the time of the failure. This is a strong indication that the failure happens solely on the client-side. I cannot make sense of it yet though:

2013-09-25 18:29:40,428 ERROR [main] (org.rhq.bindings.client.ResourceClientFactory)- Could not create custom interface for resource with id 10105
java.lang.RuntimeException: org.rhq.bindings.client.dynamic.ResourceClientProxyc453513b888ed929a98a263964ebcade00c58f523e2ea5a8e2c65d4f9f58a385: frozen class (cannot edit)
        at javassist.ClassPool.checkNotFrozen(ClassPool.java:568)
        at javassist.ClassPool.makeInterface(ClassPool.java:790)
        at javassist.ClassPool.makeInterface(ClassPool.java:775)
        at org.rhq.bindings.client.ResourceClientFactory.defineCustomInterface(ResourceClientFactory.java:149)
        at org.rhq.bindings.client.ResourceClientFactory.getResource(ResourceClientFactory.java:79)
        at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.org.mozilla.javascript.internal.MemberBox.invoke(MemberBox.java:167)
        at sun.org.mozilla.javascript.internal.NativeJavaMethod.call(NativeJavaMethod.java:245)
        at sun.org.mozilla.javascript.internal.Interpreter.interpretLoop(Interpreter.java:1706)
        at sun.org.mozilla.javascript.internal.Interpreter.interpret(Interpreter.java:849)
        at sun.org.mozilla.javascript.internal.InterpretedFunction.call(InterpretedFunction.java:162)
        at sun.org.mozilla.javascript.internal.ContextFactory.doTopCall(ContextFactory.java:433)
        at com.sun.script.javascript.RhinoScriptEngine$1.superDoTopCall(RhinoScriptEngine.java:116)
        at com.sun.script.javascript.RhinoScriptEngine$1.doTopCall(RhinoScriptEngine.java:109)
        at sun.org.mozilla.javascript.internal.ScriptRuntime.doTopCall(ScriptRuntime.java:3161)
        at sun.org.mozilla.javascript.internal.InterpretedFunction.exec(InterpretedFunction.java:173)
        at sun.org.mozilla.javascript.internal.Context.evaluateReader(Context.java:1159)
        at com.sun.script.javascript.RhinoScriptEngine.eval(RhinoScriptEngine.java:214)
        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:249)
        at org.rhq.enterprise.client.commands.ScriptCommand.executeScriptFile(ScriptCommand.java:287)
        at org.rhq.enterprise.client.commands.ScriptCommand.execute(ScriptCommand.java:108)
        at org.rhq.enterprise.client.ClientMain$StartupConfiguration.process(ClientMain.java:141)
        at org.rhq.enterprise.client.ClientMain.main(ClientMain.java:161)

Comment 4 Lukas Krejci 2013-09-25 20:14:18 UTC
This is a duplicate of BZ 829944, which was fixed in RHQ 4.5.0 by commit a48f5cda6610222c37b42fb80d29fa589a864ab5.

Comment 5 Lukas Krejci 2013-09-25 20:20:36 UTC
Moving to ON_QA as JON 3.2.0 builds already have the fix.

Comment 6 Filip Brychta 2013-11-07 16:23:28 UTC
Verified on
Version :	
3.2.0.ER5
Build Number :	
2cb2bc9:225c796

Unable to reproduce on 
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.6) (fedora-71.1.11.6.fc16-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

for both remote CLI and local CLI.