Bug 1615974

Summary:	[RFE][Tracker] - Replace fluentd for shipping metrics and log, switch to rsyslog.
Product:	[oVirt] ovirt-engine-metrics	Reporter:	Yaniv Lavi <ylavi>
Component:	RFEs	Assignee:	Shirly Radco <sradco>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Ivana Saranova <isaranov>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	1.1.6.1	CC:	aperotti, bugs, dfediuck, fgarciad, guillaume.pavese, lsvaty, mkalinin, myllynen, rdlugyhe, sbonazzo, sradco
Target Milestone:	ovirt-4.3.1	Keywords:	BuildBlocker, FutureFeature
Target Release:	---	Flags:	rdlugyhe: needinfo- rule-engine: ovirt-4.3+ rule-engine: blocker+ lsvaty: testing_plan_complete? ylavi: planning_ack+ rule-engine: devel_ack+ lsvaty: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	The current release replaces Fluentd with Rsyslog, which can collect oVirt logs, engine.log, VDSM logs, and collectd metrics. Systems upgraded from 4.2 will still have Fluentd installed, but it will be disabled and stopped. After upgrading to 4.3, you can remove the Fluentd packages. Fluentd will not be supported in RHEL 8. Rsyslog offers better performance. Rsyslog can output to Elasticsearch on Red Hat OpenShift Container Platform. Sending data to your own instance of Elasticsearch is not currently supported. Collectd is reconfigured to use write_syslog, a new plugin, to send metrics to Rsyslog. When deploying ovirt metrics, Rsyslog is configured on the Red Hat Virtualization Manager and host to collect and ship the data to the requested target.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-03-13 16:40:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Metrics	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1627753, 1627756, 1628622, 1629437, 1630263, 1650972, 1651588, 1665072, 1665073, 1678180
Bug Blocks:	1647205

Description Yaniv Lavi 2018-08-14 16:02:40 UTC

Description of problem:
Need to drop fluentd as upstream has stopped being maintained and performance has not been good. We want to replace it with rsyslog for logs and PCP for metrics shipping from collectd.

The same interface for the metrics role would be maintained.

Upgrade should be using host deploy or using the role directly.

Comment 1 Shirly Radco 2018-09-05 13:14:09 UTC

The deployment of logging and metrics will be based on the linux-system-roles:
https://github.com/linux-system-roles/metrics
https://github.com/linux-system-roles/logging

Comment 2 Sandro Bonazzola 2018-09-11 11:46:53 UTC

Raising priority to urgent and marking as blocker. If this has to be done in 4.3 we need to get new deps in and old deps out as soon as possible.

Comment 3 Lukas Svaty 2018-10-11 12:36:43 UTC

Hi can we get a feature page for this?

Comment 4 Sandro Bonazzola 2019-01-16 10:45:23 UTC

Moving back to POST since some of the bugs blocking this tracker are still in NEW state.

Comment 5 Sandro Bonazzola 2019-01-28 13:56:11 UTC

not blocking oVirt 4.3.0

Comment 6 Rolfe Dlugy-Hegwer 2019-02-11 14:59:26 UTC

Hi, Shirly. 

Please tech review the contents of the Doc Text field. In particular, we need to make sure the last two paragraphs are clear to end users. (e.g., What does BYO Elasticsearch mean?) Please feel free to edit/improve the text directly. 

Thank you, 

Rolfe

Comment 7 Ivana Saranova 2019-02-20 11:54:22 UTC

Shirly, could you please provide reproduction steps for this? Thank you

Comment 8 Shirly Radco 2019-02-20 12:12:16 UTC

1. Configure engine and hosts to send metrics and logs
2. Fields and metrics names are the same as it was in 4.2.

Comment 9 Ivana Saranova 2019-03-01 12:44:13 UTC

Steps:

1. Configure engine and hosts to send metrics and logs

     /usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_machines_for_metrics.sh

2. Check that playbook run was successful
3. Check fields and metrics names in vi /etc/ovirt-engine-metrics/config.yml.d/config.yml are the same as in 4.2


Results:
Playbook run was successful with no errors.
Fields and metrics names are same as in 4.2:

```
# This is an example file for oVirt metrics configuration.
#
# To run the oVirt metrics configuration playbook, copy this file
# to /etc/ovirt-engine-metrics/config.yml.d/ directory,
# rename it to have yml suffix and edit it with your environment details.

# Environment name. Can be used to identify data collected in a single central
# store sent from more than one oVirt engine.
# Maximum field length is 49 characters.
# For more details see:
# https://www.ovirt.org/develop/release-management/features/metrics/metrics-store-installation/#ovirt-metrics-store-setup
ovirt_env_name: my-test-engine

# Address or hostname (FQDN) of the Elasticsearch server host.
fluentd_elasticsearch_host: metrics-store

# Path to preallocated partition to be used for elasticsearch persistent storage
# ovirt_elasticsearch_mounted_storage_path: /var/lib
```


Verified in: ovirt-engine-4.3.1.1-0.1.el7.noarch, ovirt-engine-metrics-1.2.0.3-1.el7ev.noarch

Comment 10 Sandro Bonazzola 2019-03-13 16:40:29 UTC

This bugzilla is included in oVirt 4.3.1 release, published on February 28th 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.