Bug 1435994 - [RFE] Configure GenericJMX collectd plugin
Summary: [RFE] Configure GenericJMX collectd plugin
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: ovirt-engine-metrics
Classification: oVirt
Component: RFEs
Version: 1.0.1
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: ---
Assignee: Shirly Radco
QA Contact: Ivana Saranova
URL:
Whiteboard:
Depends On:
Blocks: oVirt-Metrics-and-Logs
TreeView+ depends on / blocked
 
Reported: 2017-03-26 13:58 UTC by Shirly Radco
Modified: 2020-06-26 16:38 UTC (History)
10 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2020-04-01 14:48:10 UTC
oVirt Team: Metrics
Embargoed:
sradco: ovirt-4.5?
lsvaty: testing_plan_complete+
sradco: planning_ack?
ylavi: devel_ack?
lsvaty: testing_ack+


Attachments (Terms of Use)
020-ovirt-engine-jmx.conf (2.67 KB, text/plain)
2019-01-01 11:09 UTC, Roy Golan
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 82926 0 'None' ABANDONED Configure collectd jmx plugin for ovirt-engine 2020-04-16 17:25:11 UTC

Description Shirly Radco 2017-03-26 13:58:19 UTC
Description of problem:
We would like to be able to monitor the java processes.

The GenericJMX collectd plugin requires specifying the mbeans we wish to monitor.

Roy, Please provide the mbeans list we wish to monitor.

Comment 1 Roy Golan 2017-05-21 06:39:15 UTC
All the beans that are exposed are interesting. Lets see if this is too much and then we can cut some of the less interesting stuff. Here is a short breakdown:

oVirt specific
- Lock Manager "org/ovirt/engine/core/bll/lock/InMemoryLockManager"
  Useful for debugging entities which are locked
- Backend      "org/ovirt/engine/core/bll/Backend"
  The Generic API interface where both UI and REST api are tunneled through. It 
  should have some stats about hit count and so on

wildfly and jvm
- All the beans provided by jvm and wildfly application server are interesting and will supply info about JVM internals, memory, GC, thread pool utilization, DB connection pool utilization and stats and more.

Comment 2 Yaniv Kaul 2017-06-07 13:01:29 UTC
Shirly - devel-ack?

Comment 3 Fabrice Bacchella 2017-06-07 20:58:25 UTC
People using other tools than collected are interested in this kind of informations too. A documentation explaining and documenting them is welcome.

Comment 4 Yaniv Kaul 2017-06-07 21:32:28 UTC
(In reply to Fabrice Bacchella from comment #3)
> People using other tools than collected are interested in this kind of
> informations too. A documentation explaining and documenting them is welcome.

Fabrice, please look at comment 1 - anything else you need to monitor on oVirt internals?

Comment 5 Fabrice Bacchella 2017-06-07 21:41:23 UTC
I have no idea. I wanted to play with jconsole to find what to collect and start writing our probes. I planned to start with usual gc stuff.

Comment 6 Yaniv Kaul 2017-09-05 08:23:49 UTC
Shirly - what's the status of this RFE?

Comment 7 Shirly Radco 2017-09-07 11:52:27 UTC
Not sure it will make it. Will try to get it in.

Comment 8 Lukas Svaty 2017-10-10 15:19:25 UTC
Hi Shirly,

can you post any details about this feature, what samples will we collect, aggregate and send to metrics store? What are the plugins used for collection? How data is stored in samples? etc. etc.

Can I count on all data mentioned in comment#1? For RFE plan?

LS

Comment 9 Lukas Svaty 2017-11-07 11:47:51 UTC
requires_doc_text- please reconsider.
As this is not documented on ovirt wiki, update at least BZ doctext pls.

Comment 10 Lukas Svaty 2017-11-14 12:41:55 UTC
readding needinfo from comment#8

can you post any details about this feature, what samples will we collect, aggregate and send to metrics store? What are the plugins used for collection? How data is stored in samples? etc. etc.

Comment 11 Shirly Radco 2017-11-15 12:48:03 UTC
At first we will collect "memory_pool", "memory-heap", "memory-nonheap", "gc-count", "gc-time", "thread", "thread-daemon".

Comment 12 Roy Golan 2017-11-23 11:06:02 UTC
The collectd plugin isn't able to connect to wildfly jmx. To make it work Shirly you must spend time and see how does the shipped /usr/share/ovirt-engine-wildfly/bin/jconsole.sh is able to connect. I opened the bash script and saw they are adding some jars to the classpath. I added those the jmx plugin, connected to service:jmx:remote://localhost:8707 but it didn't work. So extra work needed.

What is working is the RESTful HTTP management interface (which is doing internal jmx probably). For example with one call I'm able to get all the runtime stats. A plan B maybe to use Plugin:cURL-JSON  to get that using this URL

curl "http://localhost:8706/management?include-runtime=true&recursive=true&json.pretty=1" -u admin@internal:pass

And the heap memory looks like that:

"memory" : {
                "heap-memory-usage" : {
                    "init" : 6228541440,
                    "used" : 1862858040,
                    "committed" : 5950144512,
                    "max" : 5950144512
                },
                "non-heap-memory-usage" : {
                    "init" : 2555904,
                    "used" : 197413056,
                    "committed" : 212762624,
                    "max" : -1
                },
                "object-name" : "java.lang:type=Memory",
                "object-pending-finalization-count" : 0,
                "verbose" : false
            },

Comment 13 Roy Golan 2019-01-01 11:08:04 UTC
Update:
To make it work we need to
1. add wildfly's jboss-client jar in the classpath of the plugin /usr/share/ovirt-engine-wildfly/bin/client/jboss-client.jar
  
```
  JVMARG "-Djava.class.path=/usr/share/ovirt-engine-wildfly/bin/client/jboss-client.jar:/usr/share/collectd/java/collectd-api.jar:/usr/share/collectd/java/generic-jmx.jar"
```

2. Add user and password to the collection section

```
<Connection>
      ServiceURL "service:jmx:remoting-jmx://localhost:8707"
      Host "localhost"
      User "admin@internal"
      Password "123"
...
```

3. Make sure selinux has the port declared
```
sudo semanage  port -a -t dns_port_t -p tcp 8707
```

The whole collectd conf file is shared on the bug.

Comment 14 Roy Golan 2019-01-01 11:09:09 UTC
Created attachment 1517734 [details]
020-ovirt-engine-jmx.conf

Comment 15 Roy Golan 2019-01-01 11:10:25 UTC
Shirly I think we can make progress with this.

Mordechai give it a try!

Comment 16 Roy Golan 2019-01-01 12:20:27 UTC
One thing we need to handle is the password on clear text conf file. Shirly its worth opening a bug to collectd for it.

Comment 17 Sandro Bonazzola 2019-01-28 09:40:50 UTC
This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

Comment 18 Michal Skrivanek 2020-03-19 15:42:00 UTC
We didn't get to this bug for more than 2 years, and it's not being considered for the upcoming 4.4. It's unlikely that it will ever be addressed so I'm suggesting to close it.
If you feel this needs to be addressed and want to work on it please remove cond nack and target accordingly.

Comment 19 Michal Skrivanek 2020-04-01 14:48:10 UTC
Closing old bug. Please reopen if still relevant/you want to work on it.


Note You need to log in before you can comment on or make changes to this bug.