Bug 1350350 - [RFE] Add metrics module to vdsm
Summary: [RFE] Add metrics module to vdsm
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: General
Version: 4.18.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-4.0.4
: 4.18.5
Assignee: Yaniv Bronhaim
QA Contact: Petr Kubica
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-27 08:39 UTC by Yaniv Bronhaim
Modified: 2016-09-26 12:41 UTC (History)
11 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2016-09-26 12:41:50 UTC
oVirt Team: Infra
Embargoed:
oourfali: ovirt-4.0.z?
pkubica: testing_plan_complete+
rule-engine: planning_ack?
rule-engine: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 59772 0 ovirt-4.0 MERGED Introduce reports package 2016-06-27 08:42:36 UTC
oVirt gerrit 59773 0 ovirt-4.0 MERGED Add hawkular reporter to reports 2016-06-27 08:43:21 UTC
oVirt gerrit 59774 0 ovirt-4.0 MERGED Adding report_stats to host.api 2016-06-27 09:05:59 UTC
oVirt gerrit 59775 0 ovirt-4.0 MERGED Adding report_stats to health reports 2016-06-27 09:06:36 UTC
oVirt gerrit 59776 0 ovirt-4.0 MERGED Send host reports each sampling interval from HostMonitor 2016-06-27 09:06:53 UTC
oVirt gerrit 62142 0 ovirt-4.0 NEW Collect ksm values only if exist 2016-08-09 14:09:29 UTC
oVirt gerrit 62143 0 ovirt-4.0 NEW Adding report_stats to virt.stats 2016-08-09 14:09:48 UTC
oVirt gerrit 62144 0 ovirt-4.0 NEW Adding get_batch to StatsCache - returns dict with all vms stat 2016-08-09 14:10:07 UTC
oVirt gerrit 62145 0 ovirt-4.0 NEW Send vm info report each sampling interval from VMBulkSampler 2016-08-09 14:10:26 UTC
oVirt gerrit 62146 0 ovirt-4.0 NEW Send metrics info only if metrics enabled to avoid parsing logic 2016-08-09 14:11:05 UTC
oVirt gerrit 62147 0 ovirt-4.0 MERGED Rename reports to metrics 2016-08-17 09:29:58 UTC
oVirt gerrit 62357 0 ovirt-4.0 MERGED Fix in hawkular provider 2016-08-16 11:00:06 UTC

Description Yaniv Bronhaim 2016-06-27 08:39:46 UTC
Introduce report statistics feature for ovirt hosts. In Vdsm we implement reporting mechanism directly from the hosts using statsd or hawkular python sdk implementation.

Comment 1 Red Hat Bugzilla Rules Engine 2016-06-27 08:44:05 UTC
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 2 Red Hat Bugzilla Rules Engine 2016-06-27 08:44:38 UTC
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 3 Yaniv Bronhaim 2016-08-02 08:06:14 UTC
Please rename "reports" to "metrics" module. we renamed it in the code, so better to be aligned

Comment 4 Petr Kubica 2016-08-02 08:19:47 UTC
Renamed

Comment 5 Petr Kubica 2016-08-11 11:31:37 UTC
Hi, I found a problem with hawkular reporting:
collector_type = hawkular

Traceback (most recent call last):
  File "/usr/share/vdsm/vdsm", line 149, in run
    serve_clients(log)
  File "/usr/share/vdsm/vdsm", line 88, in serve_clients
    metrics.start()
  File "/usr/lib/python2.7/site-packages/vdsm/metrics/__init__.py", line 35, in start
    _reporter.start(config.get('metrics', 'collector_address'))
  File "/usr/lib/python2.7/site-packages/vdsm/metrics/hawkular.py", line 44, in start
    concurrent.thread(_run, name='hawkular', address=address).start()
TypeError: thread() got an unexpected keyword argument 'address'

Version:
vdsm-4.18.11-1.el7ev.x86_64

Comment 6 Yaniv Bronhaim 2016-08-16 09:17:50 UTC
Hi, sorry to miss that comment for few days :\ posted a fix https://gerrit.ovirt.org/62355 , lets try to push and verify it quickly ?
without the fix setting hawkular provider doesn't work at all.

Comment 7 Petr Kubica 2016-08-17 08:46:40 UTC
Hi, all right. I tried to verify with the small change in hawkular.py (I only changed relevant row) but now I get a new exception:

hawkular::ERROR::2016-08-17 08:33:26,095::utils::374::root::(wrapper) Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 371, in wrapper
    return f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 177, in run
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/metrics/hawkular.py", line 81, in _run
    client.put(items)
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 273, in put
    self._post(self._get_metrics_raw_url(self._get_url(l)), r[l])
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 204, in _post
    self._http(url, 'POST', data)
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 191, in _http
    self._handle_error(e)
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 226, in _handle_error
    raise e
HawkularMetricsError: HTTP Error 401: <html><head><title>Error</title></head><body>Unauthorized</body></html>

I'm using hawkular in docker - I used this manual for setup: http://pilhuhn.blogspot.cz/2016/06/using-hawkular-services-via-docker.html

If you want, I can lend you my environment for check.

Comment 8 Petr Kubica 2016-08-17 12:25:18 UTC
Seems that xx.xx.xx.xx:8080/hawkular/metrics requires login & password (which are known)

Comment 9 Yaniv Bronhaim 2016-08-22 08:22:14 UTC
I worked with https://github.com/hawkular/docker-hawkular-metrics and didn't fail on authentication issues. Maybe its related to differences between hawkular and hawkular-metrics only setups

Comment 10 Petr Kubica 2016-09-14 16:00:02 UTC
Verifiaction steps for hawkular:
1. unpack https://github.com/hawkular/hawkular-metrics/releases/download/0.18.0/hawkular-metrics-wildfly-embedded-cassandra-0.18.0.Final.tar.gz
2. run /bin/standalone.sh -b=0.0.0.0
3. configure metrics module in /etc/vdsm.conf (and install packages for hawkular "pip install hawkular-client")
4. restart vdsmd service
5. check http://$SERVER:8080/hawkular/metrics/gauges with http header "Hawkular-Tenant: oVirt"
6. You should see metrics from host vdsm in json format
7. You can use Graphana for nicer displaying metrics

Verifiaction steps for statsd:
1. Have statsd and graphite environment
2. configure metrics module in /etc/vdsm.conf (and install packages for hawkular "pip install statsd")
3. restart vdsmd service
4. Check metrics in Graphite 

Verified in vdsm-4.18.11-1.el7ev.x86_64
Authentication issue in hawkular case will be tracked in separated bug. 
https://bugzilla.redhat.com/show_bug.cgi?id=1376087


Note You need to log in before you can comment on or make changes to this bug.