Bug 1405813

Summary:	[RFE] Configure collectd and fluentd on hosts
Product:	[oVirt] ovirt-engine	Reporter:	Yedidyah Bar David <didi>
Component:	RFEs	Assignee:	Yedidyah Bar David <didi>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Lukas Svaty <lsvaty>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	---	CC:	bugs, didi, lsvaty, trichard, ylavi
Target Milestone:	ovirt-4.1.0-rc	Keywords:	FutureFeature
Target Release:	4.1.0.3	Flags:	rule-engine: ovirt-4.1+ ykaul: exception+ gklein: testing_plan_complete- ylavi: planning_ack+ sbonazzo: devel_ack+ lsvaty: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	A script is now supplied to configure collectd and fluentd on hosts to send statistics to a central store.	Story Points:	---
Clone Of:
Clones:	1410044 (view as bug list)		Environment:
Last Closed:	2017-04-27 09:36:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1392784, 1405809, 1414277, 1418985
Bug Blocks:	1402901, 1410044, 1451625

Description Yedidyah Bar David 2016-12-18 15:42:55 UTC

Description of problem:

Once a user sets up a metrics store (bug 1405812), and configures the engine to use them (bug 1405809), a cron job should automatically configure all active hosts with compatibility level 4.1 to send metrics to this store. Current plan is to do this using ansible by enhancing:

https://github.com/centos-opstools/opstools-ansible

Comment 1 Yedidyah Bar David 2017-01-23 14:50:10 UTC

Note for QE/Doc. To test metrics:

1. Install and configure fluentd, elasticsearch and kibana on some machine. Currently it should be a new machine, separate from the engine and the hosts. This will be called "the metrics store machine". See bug 1405812 for more info. You can use [1] for now. Current plan is to eventually use OCP with common logging, not sure about docs for that. You can find example configuration files for fluentd in [2].

2. Install and setup an engine.

3. Install and activate one or more 4.1 hosts.

4. On the engine machine, create a directory /etc/ovirt-engine/metrics.

5. Copy the ca certificate used in the metrics store machine to /etc/ovirt-engine/metrics/fluentd_ca_cert.pem on the engine machine.

6. Create on the engine machine a file /etc/ovirt-engine/metrics/config.yml with the following two lines:
fluentd_fluentd_host: my.fluentd.host
fluentd_shared_key: my_shared_key
Replace "my.fluentd.host" with the fqdn of the metrics store machine, and "my_shared_key" with the shared key configured in fluentd on the metrics store machine.

7. Run on the engine machine, as root:
/usr/share/ovirt-engine/setup/ansible/configure_ovirt_hosts_for_metrics.sh

It runs ansible and should finish without errors.
When finished, you should be able to see statistics in the kibana console about your hosts, VMs, etc.

We should probably have more detailed documentation for this feature. Once written, it should appear in [3].

[1] https://sradcoblog.wordpress.com/2016/07/19/ovirt-metrics-elk/
[2] https://github.com/sradco/ovirt-metrics-conf/tree/master/metrics-store
[3] http://www.ovirt.org/develop/release-management/features/engine/metrics-store/

Comment 2 Yedidyah Bar David 2017-01-23 14:52:35 UTC

See previous comment for some documentation. Setting "require doc text" to "-" because I guess we need a proper doc bug for this feature.

Comment 3 Lukas Svaty 2017-02-03 11:10:11 UTC

two of the dependant bugs are not fixed yet, moving back to ASSIGNED

Comment 4 Red Hat Bugzilla Rules Engine 2017-02-03 11:10:17 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Yedidyah Bar David 2017-02-06 15:38:20 UTC

Moving back to QE. One dependent bug closed, other moved to QE as well.

Comment 6 Yedidyah Bar David 2017-02-14 13:12:29 UTC

Setting needinfo on myself to move this bug to a new product once we create one, for https://gerrit.ovirt.org/#/admin/projects/ovirt-metrics .

Comment 7 Red Hat Bugzilla Rules Engine 2017-02-14 13:12:35 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 8 Yaniv Lavi 2017-02-15 11:52:29 UTC

Why is this back to assigned?

Comment 9 Yedidyah Bar David 2017-02-15 12:45:07 UTC

(In reply to Yaniv Dary from comment #8)
> Why is this back to assigned?

See comment 6. But I agree it does not make much sense. Moving back to QE. Created bug 1422505 for that.

Comment 10 Yedidyah Bar David 2017-03-02 08:19:49 UTC

An update:

We decided to split this off the engine.

Implications for developers:

There is now a new gerrit.ovirt.org repo called 'ovirt-engine-metrics'.

Implications for everyone, also updating comment 1:

There is a new package called ovirt-engine-metrics. It replaces the package ovirt-engine-hosts-ansible-inventory . On upgrade from 4.1.0, this should happen automatically.

Configuration for this is now in /etc/ovirt-engine-metrics .

It already provides there a file config.yml.example, which you can copy to config.yml and edit.

This file should also include the ca cert path, which is now not required to be copied there to a specific name.

You should then run:

/usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_hosts_for_metrics.sh

Comment 11 Lukas Svaty 2017-03-29 12:55:33 UTC

verified with ovirt-engine-metrics-1.0.0-1.el7ev.noarch