Bug 1025023

Summary: Add configuration option to enable/disable Infinispan statistics
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Andrig T Miller <anmiller>
Component: ClusteringAssignee: Radoslav Husar <rhusar>
Status: CLOSED CURRENTRELEASE QA Contact: Jitka Kozana <jkudrnac>
Severity: unspecified Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.0.0, 6.0.1, 6.1.0, 6.0.2 (SOA), 6.2.0, 6.1.1, 6.2.1, 6.2.2, 6.2.3, 6.2.4CC: kkhan, myarboro, paul.ferraro, rachmato, rhusar, rjanik, rsvoboda, sgilda, smcgowan, smumford
Target Milestone: DR5   
Target Release: EAP 6.3.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
This release of EAP 6 includes an enhancement to the clustering system. The Infinispan subsystem can automatically gather usage statistics. It is possible to enable or disable statistics explicitly per cache container and per cache. These statistics can help profile and debug applications and caches, however their use incurs a small overhead. They are disabled by default and can be enabled if needed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-28 15:42:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andrig T Miller 2013-10-30 18:34:50 UTC
Description of problem:

We see the Infinispan statistics as a hotspot when profiling for performance.

Version-Release number of selected component (if applicable):


How reproducible:

All the time.

Steps to Reproduce:
1.
2.
3.

Actual results:

Throughput suffers because of the statistics.

Expected results:

Be able to turn statistics off through the subsystem configuration.

Additional info:

Comment 1 JBoss JIRA Server 2013-11-05 20:42:15 UTC
Paul Ferraro <paul.ferraro> made a comment on jira WFLY-2415

PR will disable statistics by default and adds a configuration option to enable them per cache and cache-container.

Comment 3 Kabir Khan 2014-02-27 17:12:16 UTC
Merged and acked on QA's behalf. This is for a RFE: https://issues.jboss.org/browse/EAP6-76

Comment 4 Richard Janík 2014-03-11 12:00:40 UTC
It seems that per cache setting is not currently possible, this is what was tested:

I had a server instance with an application deployed and no changes to standalone-ha.xml (server started with this profile). Connecting to the server with jconsole showed that statistics are correctly not taken by default. (thus, value "undefined" for statistics-enabled behaves as "false", that's correct)

Changing the standalone-ha.xml or modifying the settings through cli had these outcomes (web container, repl cache for example):

cache-container cache outcome
---------------|-----|-------
false           true  the cache that had statistics-enabled set to true still
                      didn't collect any statistics
false           false no cache collected any statistics, correct
true            true  statistics were collected for all caches of the given 
                      cache container
true            false statistics were collected for all caches of the given
                      cache container regardless of per cache setting

Thus, I couln't enable collecting statistics per cache. When reading attributes through jboss-cli it always shows the correct values (true/false/undefined for cache/cache-container), it seems that setting per cache is not propagated to mbeans though. When connecting to the server with jconsole and looking at the mbeans, their settings only reflects the setting of a cache-container.

Btw, this is also how Wildfly 8.0.0.Final behaves.

Comment 5 Radoslav Husar 2014-03-12 14:59:15 UTC
I have verified that the setting is sucessfully applied to org.infinispan.configuration.cache.JMXStatisticsConfigurationBuilder hinting at a possible Infinispan bug.

Comment 6 Richard Achmatowicz 2014-03-12 19:00:36 UTC
I did this:
- start wildfly 
- start JBoss CLI
- start JConsole and look at MBeans for cache managers and caches
-- check if cache manager stats are enabled by checking CacheManager MBean attribute global configuration as property string
-- check if cache stats are enabled by checking Statistics MBean attribute statisticsEnabled

If found these combinations in the managenment CLI produced the correct results in the corresponding MBeans:
- cache-container=false, cache=false
- cache-container=true, cache=true
and these combinations did not:
- cache-container=false, cache=true
- cache-container=true, cache=false

This same stat of affairs happens whether we use a :reload from the CLI or we use a reboot of the server from the command line between changes to the management attributes.

Comment 7 Richard Achmatowicz 2014-03-12 20:36:10 UTC
It's a Wildfly problem: see https://issues.jboss.org/browse/WFLY-3106

Comment 11 Richard Janík 2014-03-20 13:32:26 UTC
Looks good now.

The cache/cache-manager combinations mentioned above now work correctly. There is one thing though - "undefined" state is now entirely equivalent to "false". ("undefined" = <cache-container name="a" ... > , "false" = <cache-container name="a" ... statistics-enabled="false">)

What that means is that you won't be able to enable statistics for any caches only by setting cache-manager to statistics-enabled="true" because you'll still have to explicitly set statistics-enabled to "true" for the caches. Cache manager setting still enables some other statistics, so it's functionality isn't reduced to zero. This seems a little unintuitive to me, but since I don't know of any official documentation and I can enable statistics for any subset of caches, I'll verify it.

(I'd expect "undefined" setting for cache to be overridden by "true" setting for cache manager but behave as "false" otherwise.)

Summary:
VERIFIED.
Is there a documentation that explains how exactly this should behave?

Comment 12 Radoslav Husar 2014-06-30 12:03:55 UTC
Documentation: this is a customer drive feature request that we implemented. This should be documented so that customers know about it and can leverage this new option. If anyway wants to document this further, see [1] where I explained what are the combinations and which take precedence in terms of cache container and cache settings.

[1] https://issues.jboss.org/browse/WFLY-3377?focusedCommentId=12969081&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12969081

Comment 13 sgilda 2014-07-29 20:43:03 UTC
Modified the release notes text per comment in bug 1123334.