Bug 1371530 - [RFE][Metrics Store] Install Collectd and fluentd with relevant plugins
Summary: [RFE][Metrics Store] Install Collectd and fluentd with relevant plugins
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-host-deploy
Classification: oVirt
Component: RFEs
Version: master
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ovirt-4.1.0-beta
: 1.6.0
Assignee: Yedidyah Bar David
QA Contact: Lukas Svaty
URL:
Whiteboard:
: 1371524 (view as bug list)
Depends On: 1371524 1403865
Blocks: 1373101 1379403 oVirt-Metrics-and-Logs
TreeView+ depends on / blocked
 
Reported: 2016-08-30 12:16 UTC by Shirly Radco
Modified: 2017-03-09 07:46 UTC (History)
9 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-02-15 14:53:32 UTC
oVirt Team: Integration
Embargoed:
rule-engine: ovirt-4.1+
rule-engine: exception+
pstehlik: testing_plan_complete-
ylavi: planning_ack+
sbonazzo: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 68469 0 master MERGED metrics: Install collectd and plugins 2016-12-14 15:32:45 UTC
oVirt gerrit 68497 0 ovirt-host-deploy-1.6 MERGED metrics: Install collectd and plugins 2016-12-15 07:04:34 UTC
oVirt gerrit 68498 0 master MERGED metrics: Install fluentd and plugins 2016-12-15 07:28:39 UTC
oVirt gerrit 68500 0 ovirt-host-deploy-1.6 MERGED metrics: Install fluentd and plugins 2016-12-15 07:48:21 UTC
oVirt gerrit 68514 0 master MERGED metrics: Install collectd-disk 2016-12-15 09:52:05 UTC
oVirt gerrit 68515 0 ovirt-host-deploy-1.6 MERGED metrics: Install collectd-disk 2016-12-15 10:09:45 UTC
oVirt gerrit 70311 0 master MERGED packaging: spec: require newer ovirt-host-deploy 2017-01-13 16:31:52 UTC
oVirt gerrit 70312 0 ovirt-engine-4.1 MERGED packaging: spec: require newer ovirt-host-deploy 2017-01-13 18:45:28 UTC

Description Shirly Radco 2016-08-30 12:16:47 UTC
Description of problem:

As part of the the metrics store solution we will need to have relevant Collectd packages installed during host deploy.

Comment 1 Sandro Bonazzola 2016-09-01 08:08:52 UTC
Moving to ovirt-host-deploy.
Didi, if collectd needs to be packaged, please open a separate bug on ovirt-distribution component.

Comment 2 Yaniv Lavi 2016-11-07 10:33:14 UTC
*** Bug 1371524 has been marked as a duplicate of this bug. ***

Comment 3 Jiri Belka 2016-11-16 13:23:09 UTC
What is currect status? Will this be part of 4.1?

Comment 4 Yaniv Kaul 2016-12-01 14:13:22 UTC
(In reply to Jiri Belka from comment #3)
> What is currect status? Will this be part of 4.1?

Yes.

Comment 5 Yedidyah Bar David 2016-12-12 15:17:30 UTC
We decided to only install, during host-deploy, and configure separately - using ansible. Changing summary accordingly.

The list of packages we need:

collectd
collectd-netlink
collectd-virt
collectd-write_http
fluentd
rubygem-fluent-plugin-rewrite-tag-filter
rubygem-fluent-plugin-secure-forward

Comment 6 Yedidyah Bar David 2016-12-14 15:36:54 UTC
Still missing fluentd and plugins

Comment 7 Lukas Svaty 2016-12-14 15:52:51 UTC
Thus you wanna change this to track only collectd and create another bug or wait for the fluentd and plugins pathces?

Comment 8 Yedidyah Bar David 2016-12-15 14:03:56 UTC
Moving to MODIFIED, hopefully that's enough.

collectd-disk was split into its own package only very recently. If other plugins will go the same path, we'll have to add their packages in later patches/bugs.

Comment 9 Lukas Svaty 2016-12-16 14:13:18 UTC
So if i understand correctly this "RFE" is just about dependency of engine on these packages so they would be installed.

Or are we setting them up some way?

Anything specific that should be caught within the testplan?

Comment 10 Yedidyah Bar David 2016-12-18 08:13:02 UTC
The current plan is:

1. Install collectd/fluentd + plugins on hosts by host-deploy. That's current bug.
2. Have them updated by the update manager.
3. Have a central metrics store machine, running elasticsearch and fluentd, installed and managed using ocp (openshift). This is handled by the Common Logging team.
4. Provide some means for the user to tell the engine where this metrics store machine is. This can be done using engine-config, and be stored in vdc_options, still no final decision AFAIK.
5. Once the above is ready, configure the hosts using ansible. I am currently working on this one, also using [1] and [2].

I am not aware of open bugs for any of the above except for current bug. Shirly - are there any? Is there anything else I am missing? Please open relevant bugs as needed. Thanks.

[1] https://github.com/centos-opstools/opstools-ansible
[2] https://gerrit.ovirt.org/66999

Comment 11 Yedidyah Bar David 2016-12-18 09:03:48 UTC
Yaniv - please see comment 10. Please open bugs as needed. For (4.) please decide how it should behave, and what values we need to store.

Ee do want to use secure forwarding. Current fluentd plugin has several different ways to do this. We need to decide what we want, and perhaps as a result add more values to allow setting/storing. Specifically, if we want to use on the hosts something like:

<match **>
  @type secure_forward
  shared_key $METRICS_STORE_SECRET
  self_hostname $HOSTNAME
  secure true
  ca_cert_path $METRICS_STORE_CACERT_PATH

  <server>
    host $METRICS_STORE_HOST
    port $METRICS_STORE_PORT # default 24284
  </server>
</match>

Then we need:

1. $METRICS_STORE_HOST

2. $METRICS_STORE_PORT

3. $METRICS_STORE_SECRET - needs to be stored securely, need to discuss this

4. $METRICS_STORE_CACERT_PATH - Basically 3 options:

4.1. Enforce using the engine CA. Then we do not need to ask for this, and the hosts will already have certs signed by it, but the user will have to make fluentd on the metrics store accept this.

4.2. Enforce using the CA on the metrics store. IIUC OCP already has such a thing built in, so will probably be easy to configure fluentd to use it, but then the user will have sign with it all the hosts' certs.

4.3. Use a 3rd party CA. Then user has to do everything - both on the metrics store and on the hosts.

4.4. Allow the user to choose. If so, we should know this and behave accordingly - if using the engine CA, we can use it, but if different, the user has to do stuff on the hosts before ansible configures them.

5. Turn on/off - perhaps we want the user to turn this on/off. "This" is the cron job that will configure the hosts using ansible, not actual metrics forwarding. If you want to allow on/off for metrics forwarding, that's another bug.

HOSTNAME will be set by ansible

There are other, more complex options, such as using different user/password pairs for different hosts.

Comment 12 Yaniv Lavi 2016-12-18 15:08:12 UTC
We agreed that the engine config tool will contain these parameters values and that there will be a value on if to use the engine CA or it is externally provided. By default the engine CA will be used.

Comment 13 Yedidyah Bar David 2016-12-19 07:12:42 UTC
Opened various bugs, you can see details by checking dependencies of the tracker bug 1402901.

Comment 14 Sandro Bonazzola 2017-01-13 09:20:11 UTC
Didi are you sure no doc-text is needed for this RFE?

Comment 15 Lukas Svaty 2017-02-03 09:08:41 UTC
both are installed / updated by package manager on host update, thus verified in rhevm-4.1.0.2-0.1.el7.noarch

configuration and deployment handled by BZ#1405813

Comment 16 Yedidyah Bar David 2017-02-14 13:06:26 UTC
(In reply to Sandro Bonazzola from comment #14)
> Didi are you sure no doc-text is needed for this RFE?

Not sure, wrote something anyway.

Comment 17 Yedidyah Bar David 2017-03-05 06:28:37 UTC
Emma, considering that in 4.1 we do not have yet "_a_ central metrics store", perhaps change to something like:

In this release, when installing or reinstalling hosts, the collectd and fluentd packages are now installed, including the relevant plugins. These can be used to send statistics to a central metrics store.


Note You need to log in before you can comment on or make changes to this bug.