Bug 913672 (PRODMGT-277, PRODMGT-409)

Summary: Add support for EJB response times in EAP6
Product: [JBoss] JBoss Operations Network Reporter: Charles Crouch <ccrouch>
Component: Plugin -- JBoss EAP 6Assignee: Heiko W. Rupp <hrupp>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: medium    
Version: JON 3.2CC: hbrock, hrupp, lkrejci, loleary, lzoubek, myarboro, theute
Target Milestone: ER07Keywords: FutureFeature
Target Release: JON 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 998645 (view as bug list) Environment:
Last Closed: 2014-01-02 20:35:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 911117    
Bug Blocks: 998645    

Description Charles Crouch 2013-02-21 18:50:56 UTC
When https://bugzilla.redhat.com/show_bug.cgi?id=911117 is completed we should update our EAP6 plugin to take advantage

This is coming from https://issues.jboss.org/browse/PRODMGT-277

Comment 1 Charles Crouch 2013-02-21 18:51:49 UTC
Obviously this is only feasible in 3.2 if its already available in a released version of EAP6

Comment 2 Charles Crouch 2013-02-21 18:55:18 UTC
From RFE review meeting with larry and alan:

"Engineering will create a BZ to track this and potentially get it into 3.2 but we won't add this to the PRD as a requirement and the feature will be implemented if time permits (and EAP provides the capability)."

Comment 3 Charles Crouch 2013-04-11 21:15:46 UTC
Dropping priority since based on Comment2 this is not a high priority feature. As I said there if we have time we'll look to address it.

Comment 4 Larry O'Leary 2013-05-24 20:01:11 UTC
Looks like some of these metrics have now made it into JBoss EAP 6.1 but some data may be missing based on https://bugzilla.redhat.com/show_bug.cgi?id=911117#c8.

Comment 5 Lukas Krejci 2013-06-06 11:21:35 UTC
EAP 6.1.0 contains the invocation statistics for individual methods on all kinds of enterprise beans.

Namely, it can report (on per-method basis):
"execution-time" - Time spend within this bean method.
"invocations" - Number of invocations processed.
"wait-time" - Time spend waiting to obtain an instance.

While investigating the metrics published by EAP, I found several that are missing from our plugin descriptor (even though they were present even in AS 7.1.1 times).

For all enterprise bean types, we do not track the following:
"peek-concurrent-invocations" - Peak concurrent invocations.
"invocations" - Number of invocations processed.
"wait-time" - Time spend waiting to obtain an instance.
"execution-time" - Time spend within a bean method.

Peek concurrent invocations seems to be like a good metric to collect (in a trendsup manner, so that we get the time derivative, which is the useful stuff). The rest of those seem to just be the aggregates of the per-method variants.

Not sure why we are not collecting these. Maybe they weren't working before, or we simply thought them not useful or just omitted them.

I will try to add them all and see what are the results with AS 7.1.1 and EAP 6.1.0.

Comment 6 Lukas Krejci 2013-09-13 12:02:46 UTC
This is in master POST RHQ 4.9.0 by BZ 998645. Thus, this will need to be manually cherry-picked over to JON 3.2.0 if it is based on RHQ 4.9.0 codebase.

Comment 7 Lukas Krejci 2013-10-07 13:16:13 UTC
Forgot to flip this over. By inspecting the git history, it turns out, this BZ is already present in JON 3.2.0.ER2, so updating accordingly.

Comment 8 Libor Zoubek 2013-11-08 14:01:46 UTC
Hello Lukas, 

I was not able to see any results for EJB3 calltimes.

Here's what I did:

I've set all metrics to 1 minutes on each SLSB directly on RHQ server resource
Then I was randomly clicking UI, running some CLI scripts to get those EJB calls. No results.

I checked EJB3 subsystem configuration in JON which shows "Enable Statistics" as unset, so I enabled it, restarted server.

Now I could see metrics on SLSBs but still no calltimes.

Reassigning back to gen more info about calltime configuration.

Comment 9 Heiko W. Rupp 2013-11-08 17:26:08 UTC
I see this in the agent log:

2013-11-08 18:10:37,306 ERROR [ResourceContainer.invoker.daemon-32] (rhq.modules.plugins.jbossas7.Ejb3BeanRuntimeComponent)- Unexpected type of results when querying method stats

Digging into what the server returns I see

JBAS014792: Unknown attribute methods

If go into the AS7-cli I see

[standalone@localhost:6999 stateless-session-bean=MeasurementDataManagerBean] :read-resource(include-runtime=true, recursive=true)
    "outcome" => "success",
    "result" => {
        "component-class-name" => "MeasurementDataManagerBean",
        "declared-roles" => [],
        "execution-time" => 0L,
        "invocations" => 0L,
        "peak-concurrent-invocations" => 0L,
        "pool-available-count" => 20,
        "pool-create-count" => 2,
        "pool-current-size" => 2,
        "pool-max-size" => 20,
        "pool-name" => "slsb-strict-max-pool",
        "pool-remove-count" => 0,
        "run-as-role" => undefined,
        "security-domain" => "other",
        "timers" => [],
        "wait-time" => 0L
[standalone@localhost:6999 stateless-session-bean=MeasurementDataManagerBean] pwd

So it may be that EAP 6.1.1 is not even exposing that info

Comment 10 Heiko W. Rupp 2013-11-08 20:12:48 UTC
Wildfly 8: 
  "stateless-session-bean" => {
                                                    "description" => "Stateless session bean component included in the deployment.",
                                                    "model-description" => {"*" => {
                                                        "description" => "Stateless session bean component included in the deployment.",
                                                        "access-constraints" => {"application" => {"deployment" => {"type" => "core"}}},
                                                        "attributes" => {
                                                            "methods" => {
                                                                "type" => OBJECT,
                                                                "description" => "Invocation metrics per method.",
                                                                "expressions-allowed" => false,
                                                                "nillable" => false,

Same in EAP6.1.1

    "result" => {
        "description" => "Stateless session bean component included in the deployment.",
        "attributes" => {
            "timers" => {
                "type" => LIST,
                "description" => "EJB timers associated with the component.",
                "expressions-allowed" => false,
                "nillable" => false,
                "value-type" => {

So EAP 6.1.1 does not support those stats.

Comment 11 Mike Foley 2013-11-08 20:43:49 UTC
per comment #10 ... the JON work is done, but there is something missing in EAP 6.1.1 that prevents these RFEs from working.  


i am:
-removing the blocks jon 3.2 GA 
-retargetting for JON 3.2.1  

i think the right status is ON_QA ... in JON 3.2.1 ... and we shall see if EAP 6.2 implements the neccessary stuff.  (ON_QA is better than ON_DEV, because the JON development work is done.)

Comment 12 Lukas Krejci 2013-11-08 21:31:17 UTC
according to BZ 911117 this should be in EAP 6.1.1.

Comment 13 Heiko W. Rupp 2013-11-09 08:44:51 UTC
right I was wrong as I was looking at the RHQ-server and not the JON-server.
EAP 6.1.1 indeed has that:

 "stateless-session-bean" => {
                                                    "description" => "Die im Deployment enthaltene stateless Session-Bean-Komponente.",
                                                    "model-description" => {"*" => {
                                                        "description" => "Die im Deployment enthaltene stateless Session-Bean-Komponente.",
                                                        "attributes" => {
                                                            "methods" => {
                                                                "type" => OBJECT,
                                                                "description" => "Aufrufmetrik pro Methode.",
                                                                "expressions-allowed" => false,
                                                                "nillable" => false,
                                                                "value-type" => {
                                                                    "execution-time" => {
                                                                        "type" => LONG,

Testing with an EAP 6.1.1 and a simple app gives:

Simple test.war with a Servlet inside, finds:
Address{path: deployment=javaee6-test-app.war,subsystem=ejb3,stateless-session-bean=SimpleStateless}, but does not have any subdeployments

But the method counters just stay empty:

[standalone@localhost:9999 stateless-session-bean=SimpleStateless] :read-resource(recursive=true,include-runtime=true)
    "outcome" => "success",
    "result" => {
        "component-class-name" => "SimpleStateless",
        "declared-roles" => [],
        "execution-time" => 0L,
        "invocations" => 0L,
        "methods" => {},

Comment 14 Libor Zoubek 2013-11-11 10:06:48 UTC
I see bug in plugin.

Now, when I know how to read EJB call info from EAP CLI, I tried kitchensink quickstart on stock EAP 6.1.1.GA

this is what I see in jboss cli

[standalone@localhost:9999 stateless-session-bean=MemberRegistration] :read-resource(include-runtime=true)
    "outcome" => "success",
    "result" => {
        "component-class-name" => "MemberRegistration",
        "declared-roles" => [],
        "execution-time" => 29L,
        "invocations" => 3L,
        "methods" => {"register" => {
            "execution-time" => 29L,
            "invocations" => 3L,
            "wait-time" => 0L
        "peak-concurrent-invocations" => 1L,
        "pool-available-count" => undefined,
        "pool-create-count" => undefined,
        "pool-current-size" => undefined,
        "pool-max-size" => undefined,
        "pool-name" => undefined,
        "pool-remove-count" => undefined,
        "run-as-role" => undefined,
        "security-domain" => "other",
        "timers" => [],
        "wait-time" => 0L

but JON (all metrics scheduled to 1m for MemberRegistration resource) shows nothing

Also .. agent log now shows:

2013-11-11 04:57:33,832 ERROR [WorkerThread#3[]] (rhq.core.pc.measurement.MeasurementManager)- Could not get measurement values
        at org.rhq.modules.plugins.jbossas7.Ejb3BeanRuntimeComponent.fillCallTimeData(Ejb3BeanRuntimeComponent.java:190)
        at org.rhq.modules.plugins.jbossas7.Ejb3BeanRuntimeComponent.getValues(Ejb3BeanRuntimeComponent.java:123)
        at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)

This eplains why I don't get metrics.

Comment 16 Heiko W. Rupp 2013-11-12 13:29:36 UTC
master e0bd38f7102141f7
release branch 3b9b7b521705

Comment 17 Lukas Krejci 2013-11-13 16:32:38 UTC
commit c948124e8a681b903c686d96851bbd36cbca31e9
Author: Lukas Krejci <lkrejci@redhat.com>
Date:   Wed Nov 13 16:51:57 2013 +0100

    [BZ 998645] - AS7 EJB invocation stats
    Now correctly react on server restart. We need to ignore the values from
    the previous collection in case we detect the server restarted, because
    after the server restart, the counting starts over from the zero again.
    (cherry picked from commit 3382132cfe70dd0997960a25f2d8234d9c2ec553)

Comment 18 Lukas Krejci 2013-11-14 16:18:22 UTC
commit 5395ac0e532a476d7f7fa7ec9d722813913a8955
Author: Lukas Krejci <lkrejci@redhat.com>
Date:   Thu Nov 14 17:11:19 2013 +0100

    [BZ 998645] - Don't spam the agent.log if the AS7 server doesn't support
    EJB method stats.
    (cherry picked from commit 71f90566f8fd42f09e01551d34020da33d21f60f)

Comment 19 Simeon Pinder 2013-11-19 15:47:37 UTC
Moving to ON_QA as available for testing with new brew build.

Comment 20 Simeon Pinder 2013-11-22 05:13:10 UTC
Mass moving all of these from ER6 to target milestone ER07 since the ER6 build was bad and QE was halted for the same reason.

Comment 21 Libor Zoubek 2013-11-25 11:20:12 UTC
verified on ER7