Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1350350

Summary:	[RFE] Add metrics module to vdsm
Product:	[oVirt] vdsm	Reporter:	Yaniv Bronhaim <ybronhei>
Component:	General	Assignee:	Yaniv Bronhaim <ybronhei>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Petr Kubica <pkubica>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.18.5	CC:	bazulay, bugs, gklein, lsurette, oourfali, pkubica, sbonazzo, srevivo, ybronhei, ycui, ykaul
Target Milestone:	ovirt-4.0.4	Keywords:	FutureFeature, TechPreview, ZStream
Target Release:	4.18.5	Flags:	oourfali: ovirt-4.0.z? pkubica: testing_plan_complete+ rule-engine: planning_ack? rule-engine: devel_ack+ pstehlik: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-09-26 12:41:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yaniv Bronhaim 2016-06-27 08:39:46 UTC

Introduce report statistics feature for ovirt hosts. In Vdsm we implement reporting mechanism directly from the hosts using statsd or hawkular python sdk implementation.

Comment 1 Red Hat Bugzilla Rules Engine 2016-06-27 08:44:05 UTC

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 2 Red Hat Bugzilla Rules Engine 2016-06-27 08:44:38 UTC

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 3 Yaniv Bronhaim 2016-08-02 08:06:14 UTC

Please rename "reports" to "metrics" module. we renamed it in the code, so better to be aligned

Comment 4 Petr Kubica 2016-08-02 08:19:47 UTC

Renamed

Comment 5 Petr Kubica 2016-08-11 11:31:37 UTC

Hi, I found a problem with hawkular reporting:
collector_type = hawkular

Traceback (most recent call last):
  File "/usr/share/vdsm/vdsm", line 149, in run
    serve_clients(log)
  File "/usr/share/vdsm/vdsm", line 88, in serve_clients
    metrics.start()
  File "/usr/lib/python2.7/site-packages/vdsm/metrics/__init__.py", line 35, in start
    _reporter.start(config.get('metrics', 'collector_address'))
  File "/usr/lib/python2.7/site-packages/vdsm/metrics/hawkular.py", line 44, in start
    concurrent.thread(_run, name='hawkular', address=address).start()
TypeError: thread() got an unexpected keyword argument 'address'

Version:
vdsm-4.18.11-1.el7ev.x86_64

Comment 6 Yaniv Bronhaim 2016-08-16 09:17:50 UTC

Hi, sorry to miss that comment for few days :\ posted a fix https://gerrit.ovirt.org/62355 , lets try to push and verify it quickly ?
without the fix setting hawkular provider doesn't work at all.

Comment 7 Petr Kubica 2016-08-17 08:46:40 UTC

Hi, all right. I tried to verify with the small change in hawkular.py (I only changed relevant row) but now I get a new exception:

hawkular::ERROR::2016-08-17 08:33:26,095::utils::374::root::(wrapper) Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 371, in wrapper
    return f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 177, in run
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/metrics/hawkular.py", line 81, in _run
    client.put(items)
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 273, in put
    self._post(self._get_metrics_raw_url(self._get_url(l)), r[l])
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 204, in _post
    self._http(url, 'POST', data)
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 191, in _http
    self._handle_error(e)
  File "/usr/lib/python2.7/site-packages/hawkular/metrics.py", line 226, in _handle_error
    raise e
HawkularMetricsError: HTTP Error 401: <html><head><title>Error</title></head><body>Unauthorized</body></html>

I'm using hawkular in docker - I used this manual for setup: http://pilhuhn.blogspot.cz/2016/06/using-hawkular-services-via-docker.html

If you want, I can lend you my environment for check.

Comment 8 Petr Kubica 2016-08-17 12:25:18 UTC

Seems that xx.xx.xx.xx:8080/hawkular/metrics requires login & password (which are known)

Comment 9 Yaniv Bronhaim 2016-08-22 08:22:14 UTC

I worked with https://github.com/hawkular/docker-hawkular-metrics and didn't fail on authentication issues. Maybe its related to differences between hawkular and hawkular-metrics only setups

Comment 10 Petr Kubica 2016-09-14 16:00:02 UTC

Verifiaction steps for hawkular:
1. unpack https://github.com/hawkular/hawkular-metrics/releases/download/0.18.0/hawkular-metrics-wildfly-embedded-cassandra-0.18.0.Final.tar.gz
2. run /bin/standalone.sh -b=0.0.0.0
3. configure metrics module in /etc/vdsm.conf (and install packages for hawkular "pip install hawkular-client")
4. restart vdsmd service
5. check http://$SERVER:8080/hawkular/metrics/gauges with http header "Hawkular-Tenant: oVirt"
6. You should see metrics from host vdsm in json format
7. You can use Graphana for nicer displaying metrics

Verifiaction steps for statsd:
1. Have statsd and graphite environment
2. configure metrics module in /etc/vdsm.conf (and install packages for hawkular "pip install statsd")
3. restart vdsmd service
4. Check metrics in Graphite 

Verified in vdsm-4.18.11-1.el7ev.x86_64
Authentication issue in hawkular case will be tracked in separated bug. 
https://bugzilla.redhat.com/show_bug.cgi?id=1376087