Bug 1405812 - [RFE] Allow easy integration with installed elasticsearch and fluentd
Summary: [RFE] Allow easy integration with installed elasticsearch and fluentd
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-distribution
Classification: oVirt
Component: General
Version: ---
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ovirt-4.2.0
: 4.2.0
Assignee: Shirly Radco
QA Contact: Lukas Svaty
URL:
Whiteboard:
Depends On:
Blocks: oVirt-Metrics-and-Logs
TreeView+ depends on / blocked
 
Reported: 2016-12-18 15:36 UTC by Yedidyah Bar David
Modified: 2019-04-28 13:53 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-20 10:46:25 UTC
oVirt Team: Metrics
Embargoed:
rule-engine: ovirt-4.2+
gklein: testing_plan_complete-
ylavi: planning_ack+
sradco: devel_ack+
lsvaty: testing_ack+


Attachments (Terms of Use)

Description Yedidyah Bar David 2016-12-18 15:36:37 UTC
Description of problem:

$Subject.

This is currently planned to be handled by common-logging, see [1]. Another option might be [2].

[1] https://github.com/openshift/openshift-ansible/tree/master/roles/openshift_hosted_logging
[2] https://github.com/centos-opstools/opstools-ansible

Comment 1 Yedidyah Bar David 2016-12-18 15:38:09 UTC
Opening this bug mainly for tracking. It might be eventually a doc bug, at least for 4.1.

Comment 2 Yaniv Kaul 2017-02-20 12:08:35 UTC
Provide infra to ask the user about:
- Fluentd server
- Certificate?
- Project name
- Elastic endpoint (if different than Fluentd server)
- Kibana URL
- Kibana credentials? (if not supporting SSO?)

Comment 3 Yedidyah Bar David 2017-02-20 12:38:15 UTC
The intention of current bug was to install, not to integrate. Since we decided that we'll rely on ocp+cl for this, we should probably either change some fields above to let them own it, or, if they track their work using other means, add a link there.

For integration, at the time, I filed bug 1405809. Eventually we decided to require the user to set these params manually, as described there and in bug 1405813 comment 1.

If you want an interactive tool for this, it might make sense to open a new bug for it.

(In reply to Yaniv Kaul from comment #2)
> Provide infra to ask the user about:
> - Fluentd server
> - Certificate?
> - Project name
> - Elastic endpoint (if different than Fluentd server)

IIUC the engine needs to know about these, in 4.2, as it needs to connect there.
So we probably need to restart it when configuring. So it might make sense to add this as a plugin to engine-setup, not a separate tool.

> - Kibana URL

I guess we might want/need this at some point to add links from the web admin to there, to show stats about entities etc., similarly to what we had with Reports. Do we already have a bz for this?

> - Kibana credentials? (if not supporting SSO?)

I think that's a different discussion. If/when we add links to kibana in the ui, and do not have SSO, we should probably allow the user to decide whether to provide creds during this point (engine-setup) and have them saved on engine machine, and having all users use same creds, or to provide creds from the web interface, which will be saved in the browser/session. Or some other means.

Comment 4 Yaniv Kaul 2017-02-20 15:23:23 UTC
(In reply to Yedidyah Bar David from comment #3)
> The intention of current bug was to install, not to integrate. Since we
> decided that we'll rely on ocp+cl for this, we should probably either change
> some fields above to let them own it, or, if they track their work using
> other means, add a link there.
> 
> For integration, at the time, I filed bug 1405809. Eventually we decided to
> require the user to set these params manually, as described there and in bug
> 1405813 comment 1.

So we either need some nicer interface or leave as is - but we are not planning to install ocp+cl right now. We may bring an Ansible script that does that, for example, as VMs on top of RHV.

> 
> If you want an interactive tool for this, it might make sense to open a new
> bug for it.
> 
> (In reply to Yaniv Kaul from comment #2)
> > Provide infra to ask the user about:
> > - Fluentd server
> > - Certificate?
> > - Project name
> > - Elastic endpoint (if different than Fluentd server)
> 
> IIUC the engine needs to know about these, in 4.2, as it needs to connect
> there.
> So we probably need to restart it when configuring. So it might make sense
> to add this as a plugin to engine-setup, not a separate tool.

This was my intention when I read this bug.

> 
> > - Kibana URL
> 
> I guess we might want/need this at some point to add links from the web
> admin to there, to show stats about entities etc., similarly to what we had
> with Reports. Do we already have a bz for this?

No, not yet.

> 
> > - Kibana credentials? (if not supporting SSO?)
> 
> I think that's a different discussion. If/when we add links to kibana in the
> ui, and do not have SSO, we should probably allow the user to decide whether
> to provide creds during this point (engine-setup) and have them saved on
> engine machine, and having all users use same creds, or to provide creds
> from the web interface, which will be saved in the browser/session. Or some
> other means.

Right. I'd like to believe SSO is a good path to pursue, though. Keeping credentials is annoying, as it's a copy of them, and you need to update them, etc.

Comment 5 Yaniv Kaul 2017-06-06 21:20:17 UTC
Yaniv - please provide the requirement for the integration.

Comment 6 Yaniv Lavi 2017-06-11 13:07:03 UTC
(In reply to Yaniv Kaul from comment #5)
> Yaniv - please provide the requirement for the integration.

I see this as a configuration option for the host metrics agents via the UI\REST.
Allow setting different parameters that will be used by the engine side Ansible script.

Comment 7 Yaniv Kaul 2017-06-11 13:55:20 UTC
(In reply to Yaniv Lavi from comment #6)
> (In reply to Yaniv Kaul from comment #5)
> > Yaniv - please provide the requirement for the integration.
> 
> I see this as a configuration option for the host metrics agents via the
> UI\REST.

We don't have configuration options exposed via UI and/or REST (for example, none of the engine-config values are)

> Allow setting different parameters that will be used by the engine side
> Ansible script.

They are in a standard config file which is easily modifiable via Ansible.

Comment 8 Yaniv Lavi 2017-06-13 08:18:47 UTC
(In reply to Yaniv Kaul from comment #7)
> (In reply to Yaniv Lavi from comment #6)
> > (In reply to Yaniv Kaul from comment #5)
> > > Yaniv - please provide the requirement for the integration.
> > 
> > I see this as a configuration option for the host metrics agents via the
> > UI\REST.
> 
> We don't have configuration options exposed via UI and/or REST (for example,
> none of the engine-config values are)
> 
> > Allow setting different parameters that will be used by the engine side
> > Ansible script.
> 
> They are in a standard config file which is easily modifiable via Ansible.

I feel conformable with this as a v1, v1.x can have a Ansible interactive mode that will give wizard experience that we can put into Cockpit as well.

Comment 9 Yaniv Kaul 2017-06-19 11:36:32 UTC
Shirly, please devel-ack based on above v1 requirement (Ansible script, YAML config file based configuration).

Comment 10 Yaniv Lavi 2017-07-31 10:58:59 UTC
Should this be MODIFIED?

Comment 11 Shirly Radco 2017-08-07 10:42:32 UTC
(In reply to Yaniv Kaul from comment #9)
> Shirly, please devel-ack based on above v1 requirement (Ansible script, YAML
> config file based configuration).

I'm moving to modified based on the v1 requirements, Ansible script, YAML config file based configuration.

Comment 12 Lukas Svaty 2017-09-21 16:15:27 UTC
verified in ovirt-engine-metrics-1.1.1-0.0.master.20170919065728.el7.centos.noarch

configuration file:
-rw-r--r--. 1 root root 895 Sep 21 16:27 /etc/ovirt-engine-metrics/config.yml

playbooks to connect to metrics store on top of fluentd:
-rwxr-xr-x. 1 root root  145 Sep 19 08:57 /usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_hosts_for_metrics.sh
-rwxr-xr-x. 1 root root 2282 Sep 19 08:57 /usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_machines_for_metrics.sh

Comment 13 Sandro Bonazzola 2017-12-20 10:46:25 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.