Bug 1475900 - [RFE] Collect the logs from vdsm.log to the metrics store
[RFE] Collect the logs from vdsm.log to the metrics store
Product: ovirt-engine-metrics
Classification: oVirt
Component: RFEs (Show other bugs)
Unspecified Unspecified
high Severity low
: ovirt-4.1.8
: ---
Assigned To: Shirly Radco
Lukas Svaty
: FutureFeature
Depends On:
Blocks: 1475135 1513032 oVirt-Metrics-and-Logs 1513703
  Show dependency treegraph
Reported: 2017-07-27 10:57 EDT by Shirly Radco
Modified: 2017-12-11 11:30 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Collect the logs from vdsm.log to the metrics store Reason: In order to allow the users a common logging where they can explore both vdsm and engine logs. Result: vdsm.log is now collected to the metric store from each host.
Story Points: ---
Clone Of:
Last Closed: 2017-12-11 11:30:10 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Metrics
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑4.1+
rule-engine: ovirt‑4.2+
lsvaty: testing_plan_complete+
ylavi: planning_ack+
rule-engine: devel_ack+
pstehlik: testing_ack+

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 80010 master MERGED fluentd: collect vdsm.log 2017-11-09 07:54 EST
oVirt gerrit 84011 master MERGED Update collection to collect vdsm.log by default 2017-11-13 07:42 EST
oVirt gerrit 84162 ovirt-engine-metrics-4.1 MERGED fluentd: collect vdsm.log 2017-11-15 06:17 EST
oVirt gerrit 84163 ovirt-engine-metrics-4.1 MERGED Update collection to collect vdsm.log by default 2017-11-15 06:17 EST

  None (edit)
Description Shirly Radco 2017-07-27 10:57:23 EDT
Description of problem:
Collect the logs from vdsm.log to the metrics store
Comment 2 Shirly Radco 2017-10-16 05:43:09 EDT
Please provide the use cases for using this log.
Comment 3 Yaniv Lavi 2017-10-29 13:04:21 EDT
(In reply to Shirly Radco from comment #2)
> Please provide the use cases for using this log.

Debugging engine-host issues in a single view.
Exploring VMs and tasks bottlenecks on the hosts.
Comment 4 Shirly Radco 2017-11-09 07:58:41 EST
configure_host_logs_processing is set to false by default.

In order to test user can run
/usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_machines_for_metrics.sh -e "configure_host_logs_processing=true"

or add 

configure_host_logs_processing: true

to the /etc/ovirt-engine-metrics/config.yml file.

When updating back to configure_host_logs_processing=false the configuration file is removed from /etc/fluentd/config.d/ and collectd is restarted.
Comment 5 Lukas Svaty 2017-11-09 08:04:35 EST
Hi Shirly, 

I do not agree with the decision of having it off by default, I see an inconsistency here between 2 collection utilities, log-collector and metrics store.

Log-collector is gathering logs from hosts by default while metrics store is not. 

Where did the decision to have it disabled by default come from? Can we change it to be enabled by default?

Comment 6 Yaniv Lavi 2017-11-09 08:11:47 EST
Since we see that system resources is not affected, let's move to be on by default.
Comment 7 Shirly Radco 2017-11-13 07:36:19 EST
I have updated the default value to true.
vdsm.log will now be collected by default.
Comment 8 Shirly Radco 2017-11-15 05:49:12 EST
Yaniv L requested this to be backport also to 4.1.
Comment 9 Lukas Svaty 2017-11-29 06:31:08 EST
Currently index for logs is not being created in ViaQ setup, thus this is blocking this verification.

More over I do not see any parsing information of vdsmd.log so it seems to me like FailedQA. Shirly any advise on verification steps here?

For engine log:
Nov 29 11:55:44 example.com fluentd[15159]: 2017-11-29 11:55:44 +0100 [info]: following tail of /var/log/ovirt-engine/engine.log

Nothing for vdsmd.log
Comment 10 Lukas Svaty 2017-11-30 09:20:15 EST
ignore the last comment, ofcourse you cant follow both engine.log and vdsm.log on a single machine as they are on different machines (host vs engine)

verified in ovirt-engine-metrics-1.0.8-1.el7ev.noarch
Comment 11 Yaniv Lavi 2017-12-03 08:24:22 EST
*** Bug 1388611 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.