Bug 1698870

Summary:	[RFE] Test sending metrics and logs to elasticsearch that is not based on OpenShift
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Shirly Radco <sradco>
Component:	Documentation	Assignee:	Shirly Radco <sradco>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Guilherme Santos <gdeolive>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.3.9	CC:	abpatil, bugs, bugzilla-qe-rhv, emarcus, gdeolive, gscott, lsurette, lsvaty, mkalinin, pelauter, rdlugyhe, srevivo, tburke
Target Milestone:	ovirt-4.3.11	Keywords:	Documentation, FutureFeature, Reopened, RFE, ZStream
Target Release:	---	Flags:	lsvaty: testing_plan_complete-
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	The Red Hat knowledge base article, "Monitoring RHV with a standalone Elasticsearch instance," has been updated, and is available at https://access.redhat.com/articles/4921101	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-01 10:02:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Metrics	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1709705

Description Shirly Radco 2019-04-11 12:04:31 UTC

Description of problem:
We need to check if the current metrics role supportsconfiguring the hosts to ship the data to elasticsearch instance that is not based on OpenShift.

Steps to Reproduce:
1. Setting up elasticsearch instance
2. Creating project.ovirt-metrics and project.ovirt-logs dynamic index templates.
3. Log in to the Manager machine using SSH.
4. Copy config.yml.example to create config.yml:
# cp /etc/ovirt-engine-metrics/config.yml.example /etc/ovirt-engine-metrics/config.yml.d/config.yml
5. Edit the ovirt_env_name and elasticsearch_host parameters in config.yml
and add the following variables:

use_omelasticsearch_cert: false
viaq_metrics_store: false

6. Save the file. These parameters are mandatory and are documented in the file.
7. Deploy Collectd and Rsyslog on the hosts:
# /usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_machines_for_metrics.sh
8. Import the pre-defined dashboards to Kibana(User will be prompt to choose the index to use for the dashboards, and he should choose the project.ovirt-metrics. and project.ovirt-logs indexes from the drop down list.

Expected results:
Data should be saved to elasticsearch and visible in Kibana.

Additional info:

Comment 1 Shirly Radco 2019-05-15 07:42:10 UTC

Additional variables that needs to be set in the config.yml file:

Set if no certificates required: 
use_omelasticsearch_cert: false

If not using https add the also following:
rsyslog_elasticsearch_usehttps_metrics: off
rsyslog_elasticsearch_usehttps_logs: off



If certificate are required user will need to specify their location,
Default locations:

# Where to find the CA certificate used to communicate with Elasticsearch
rsyslog_elasticsearch_ca_cert_path: '/etc/rsyslog.d/elasticsearch_ca_cert.pem'

# Where to find the client certificate used to communicate with Elasticsearch
rsyslog_elasticsearch_client_cert_path: '/etc/rsyslog.d/elasticsearch_client_cert.pem'

# Where to find the client certificate used to communicate with Elasticsearch
rsyslog_elasticsearch_client_key_path: '/etc/rsyslog.d/elasticsearch_client_key.pem'

Comment 2 Shirly Radco 2019-05-15 08:50:42 UTC

*** Bug 1693744 has been marked as a duplicate of this bug. ***

Comment 3 Shirly Radco 2019-05-15 08:53:57 UTC

Update to step 2 in comment #1

The dynamic index templates names should be:

project.ovirt-metrics-<ovirt_env_name>
project.ovirt-logs-<ovirt_env_name>

Comment 7 Peter Lauterbach 2019-12-02 12:34:54 UTC

Yes, we do need to support customers who have existing ELK stacks for logging and metrics. We'll come up with a tighter set of requirements for what we want to support, and a reasonable set to test against to limit the infinite field.
Let's re-open this, and track as an RFE to deliver this feature.

Comment 9 Guilherme Santos 2020-03-09 14:24:04 UTC

I was able to test it following the steps:

1. Installed ElasticSearch 5.6.16 in a fresh vm following: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/_installation.html
2. On ES machine append the following "network.host: 0.0.0.0" on /etc/elasticsearch/elasticsearch.yml
# cat "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml
3. Followed this bz description and comment #1 for metrics side
4. After deploying rsyslog and collectd, I manually edit /etc/rsyslog.d/30-elasticsearch.conf following varialbes:

# usehttps="off" (it was "on")
# allowUnsignedCerts="off" (deleted this line)

Results:
Data was being pushed to ES instance outside ocp env

Comment 10 Eli Marcus 2020-03-24 21:32:03 UTC

Hi Guilherme, sharing a draft of the KB article with you, please let me know if this provides the procedure that is needed:
 
https://docs.google.com/document/d/1pEZCIVud4a8F_qiJb17qWvQqzH-rVkWZiMYOeLdNxOo/edit?usp=sharing

Comment 11 Guilherme Santos 2020-03-27 18:52:34 UTC

It looks good to me Eli! It does provide the steps needed.

Comment 14 Guilherme Santos 2020-06-22 13:42:03 UTC

I just want to complement my comment #9 about the installation of Elastic Search.
The tar installation method, presented in the tutorial link, may have some issues depending of the environment. A better approach is to use either the repositories or rpm here: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/rpm.html#rpm-repo

Also there is a typo on step 2, the cmd should be:
# cat "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml

Comment 15 Guilherme Santos 2020-06-22 13:43:11 UTC

(In reply to Guilherme Santos from comment #14)
> I just want to complement my comment #9 about the installation of Elastic
> Search.
> The tar installation method, presented in the tutorial link, may have some
> issues depending of the environment. A better approach is to use either the
> repositories or rpm here:
> https://www.elastic.co/guide/en/elasticsearch/reference/5.6/rpm.html#rpm-repo
> 
> Also there is a typo on step 2, the cmd should be:
> # cat "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml

Typo again:
# echo "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml