Bug 1410044 - [RFE] Configure collectd and fluentd on engine
Summary: [RFE] Configure collectd and fluentd on engine
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine-metrics
Classification: oVirt
Component: RFEs
Version: 1.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.1.1-1
: ---
Assignee: Yedidyah Bar David
QA Contact: Lukas Svaty
URL:
Whiteboard:
Depends On: 1405809 1405813
Blocks: oVirt-Metrics-and-Logs 1434570
TreeView+ depends on / blocked
 
Reported: 2017-01-04 10:30 UTC by Shirly Radco
Modified: 2019-04-28 14:06 UTC (History)
6 users (show)

Fixed In Version:
Clone Of: 1405813
Environment:
Last Closed: 2017-04-21 09:46:39 UTC
oVirt Team: Integration
Embargoed:
ykaul: needinfo+
rule-engine: ovirt-4.1+
rule-engine: exception+
gklein: testing_plan_complete-
ylavi: planning_ack+
rule-engine: devel_ack+
lsvaty: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 73997 0 None None None 2017-03-20 13:04:14 UTC

Description Shirly Radco 2017-01-04 10:30:20 UTC
Description of problem:

Once a user sets up a metrics store (bug 1405812), and configures the engine to use them (bug 1405809), a cron job should automatically configure engine to send metrics to this store. Current plan is to do this using ansible by enhancing:

https://github.com/centos-opstools/opstools-ansible

Comment 1 Yedidyah Bar David 2017-02-02 14:12:48 UTC
Please provide configuration files for the engine machine.

Can't see in [1] anything for postgresql.

I guess we also do not need statsd conf, and if we want to have a common subset for both engine and hosts, also perhaps move most of builtins-conf-for-ovirt.conf.j2 to a common file and keep there only host-specific stuff.

Similarly for fluentd ovirt_host/templates/ovirt-processing.conf.

[1] github.com/sradco/ovirt-metrics-conf

Comment 2 Shirly Radco 2017-02-13 11:24:27 UTC
(In reply to Yedidyah Bar David from comment #1)
> Please provide configuration files for the engine machine.
> 
> Can't see in [1] anything for postgresql.
> 
> I guess we also do not need statsd conf, and if we want to have a common

Also virt plugin is not relevant.

> subset for both engine and hosts, also perhaps move most of
> builtins-conf-for-ovirt.conf.j2 to a common file and keep there only
> host-specific stuff.


It is possible to have to have a common file and have separate host and engine specific file. But need to keep in mind that in Fluentd the order is important.

I think we should keep it simple for now and later improve it.

> 
> Similarly for fluentd ovirt_host/templates/ovirt-processing.conf.
> 
> [1] github.com/sradco/ovirt-metrics-conf

The engine config files are at 
https://github.com/sradco/ovirt-metrics-conf/blob/master/engine/

They do not include postgresql plugin yet. But you can use them.

Comment 3 Lukas Svaty 2017-03-30 08:46:10 UTC
as discussed in private thread, this need automatic population of config for postgres plugin, waiting for new patch, moving back to ASSIGNED

Comment 4 Red Hat Bugzilla Rules Engine 2017-03-30 08:46:17 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Lukas Svaty 2017-04-04 14:33:34 UTC
removing 4.1.1 dep

Comment 6 Lukas Svaty 2017-04-07 11:53:08 UTC
[root@ls-engine1 ~]# service collectd status
Redirecting to /bin/systemctl status  collectd.service
● collectd.service - Collectd statistics daemon
   Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2017-04-07 13:23:38 CEST; 12min ago
     Docs: man:collectd(1)
           man:collectd.conf(5)
 Main PID: 2514 (collectd)
   CGroup: /system.slice/collectd.service
           └─2514 /usr/sbin/collectd -C /etc/collectd.conf

[root@ls-engine1 ~]# service fluentd status
Redirecting to /bin/systemctl status  fluentd.service
● fluentd.service - Fluentd
   Loaded: loaded (/usr/lib/systemd/system/fluentd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2017-04-07 13:23:17 CEST; 13min ago
     Docs: http://www.fluentd.org/
 Main PID: 2052 (fluentd)
   CGroup: /system.slice/fluentd.service
           ├─2052 /usr/bin/ruby /usr/bin/fluentd -c /etc/fluentd/fluent.conf
           └─2198 /usr/bin/ruby /usr/bin/fluentd -c /etc/fluentd/fluent.conf

values are being reported to kibana under ovirt.entity: engine

verified in ovirt-engine-metrics-1.0.2-1.el7ev.noarch

Yaniv:
as for  
"Yaniv Kaul 2017-04-02 03:45:02 EDT
Target Milestone: ovirt-4.1.1-1 → ovirt-4.1.2"

as this was verified  4.1.1-1 can we re-target to 4.1.1-1 thus comment#5 will be obsolete and we can re-add and track it there



rest of the engine bugs are tracked here - BZ#1429861 and BZ#1429875 are scheduled for 4.1.2


Note You need to log in before you can comment on or make changes to this bug.