Description of problem: We would like to be able to monitor the java processes. The GenericJMX collectd plugin requires specifying the mbeans we wish to monitor. Roy, Please provide the mbeans list we wish to monitor.
All the beans that are exposed are interesting. Lets see if this is too much and then we can cut some of the less interesting stuff. Here is a short breakdown: oVirt specific - Lock Manager "org/ovirt/engine/core/bll/lock/InMemoryLockManager" Useful for debugging entities which are locked - Backend "org/ovirt/engine/core/bll/Backend" The Generic API interface where both UI and REST api are tunneled through. It should have some stats about hit count and so on wildfly and jvm - All the beans provided by jvm and wildfly application server are interesting and will supply info about JVM internals, memory, GC, thread pool utilization, DB connection pool utilization and stats and more.
Shirly - devel-ack?
People using other tools than collected are interested in this kind of informations too. A documentation explaining and documenting them is welcome.
(In reply to Fabrice Bacchella from comment #3) > People using other tools than collected are interested in this kind of > informations too. A documentation explaining and documenting them is welcome. Fabrice, please look at comment 1 - anything else you need to monitor on oVirt internals?
I have no idea. I wanted to play with jconsole to find what to collect and start writing our probes. I planned to start with usual gc stuff.
Shirly - what's the status of this RFE?
Not sure it will make it. Will try to get it in.
Hi Shirly, can you post any details about this feature, what samples will we collect, aggregate and send to metrics store? What are the plugins used for collection? How data is stored in samples? etc. etc. Can I count on all data mentioned in comment#1? For RFE plan? LS
requires_doc_text- please reconsider. As this is not documented on ovirt wiki, update at least BZ doctext pls.
readding needinfo from comment#8 can you post any details about this feature, what samples will we collect, aggregate and send to metrics store? What are the plugins used for collection? How data is stored in samples? etc. etc.
At first we will collect "memory_pool", "memory-heap", "memory-nonheap", "gc-count", "gc-time", "thread", "thread-daemon".
The collectd plugin isn't able to connect to wildfly jmx. To make it work Shirly you must spend time and see how does the shipped /usr/share/ovirt-engine-wildfly/bin/jconsole.sh is able to connect. I opened the bash script and saw they are adding some jars to the classpath. I added those the jmx plugin, connected to service:jmx:remote://localhost:8707 but it didn't work. So extra work needed. What is working is the RESTful HTTP management interface (which is doing internal jmx probably). For example with one call I'm able to get all the runtime stats. A plan B maybe to use Plugin:cURL-JSON to get that using this URL curl "http://localhost:8706/management?include-runtime=true&recursive=true&json.pretty=1" -u admin@internal:pass And the heap memory looks like that: "memory" : { "heap-memory-usage" : { "init" : 6228541440, "used" : 1862858040, "committed" : 5950144512, "max" : 5950144512 }, "non-heap-memory-usage" : { "init" : 2555904, "used" : 197413056, "committed" : 212762624, "max" : -1 }, "object-name" : "java.lang:type=Memory", "object-pending-finalization-count" : 0, "verbose" : false },
Update: To make it work we need to 1. add wildfly's jboss-client jar in the classpath of the plugin /usr/share/ovirt-engine-wildfly/bin/client/jboss-client.jar ``` JVMARG "-Djava.class.path=/usr/share/ovirt-engine-wildfly/bin/client/jboss-client.jar:/usr/share/collectd/java/collectd-api.jar:/usr/share/collectd/java/generic-jmx.jar" ``` 2. Add user and password to the collection section ``` <Connection> ServiceURL "service:jmx:remoting-jmx://localhost:8707" Host "localhost" User "admin@internal" Password "123" ... ``` 3. Make sure selinux has the port declared ``` sudo semanage port -a -t dns_port_t -p tcp 8707 ``` The whole collectd conf file is shared on the bug.
Created attachment 1517734 [details] 020-ovirt-engine-jmx.conf
Shirly I think we can make progress with this. Mordechai give it a try!
One thing we need to handle is the password on clear text conf file. Shirly its worth opening a bug to collectd for it.
This bug has not been marked as blocker for oVirt 4.3.0. Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.
We didn't get to this bug for more than 2 years, and it's not being considered for the upcoming 4.4. It's unlikely that it will ever be addressed so I'm suggesting to close it. If you feel this needs to be addressed and want to work on it please remove cond nack and target accordingly.
Closing old bug. Please reopen if still relevant/you want to work on it.