Bug 1475900

Summary: [RFE] Collect the logs from vdsm.log to the metrics store
Product: [oVirt] ovirt-engine-metrics Reporter: Shirly Radco <sradco>
Component: RFEsAssignee: Shirly Radco <sradco>
Status: CLOSED CURRENTRELEASE QA Contact: Lukas Svaty <lsvaty>
Severity: low Docs Contact:
Priority: high    
Version: 1.0.5CC: bugs, lsvaty, pstehlik, sradco, syangsao, ylavi
Target Milestone: ovirt-4.1.8Keywords: FutureFeature
Target Release: ---Flags: rule-engine: ovirt-4.1+
rule-engine: ovirt-4.2+
lsvaty: testing_plan_complete+
ylavi: planning_ack+
rule-engine: devel_ack+
pstehlik: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Feature: Collect the logs from vdsm.log to the metrics store Reason: In order to allow the users a common logging where they can explore both vdsm and engine logs. Result: vdsm.log is now collected to the metric store from each host.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-11 16:30:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Metrics RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1402901, 1475135, 1513032, 1513703    

Description Shirly Radco 2017-07-27 14:57:23 UTC
Description of problem:
Collect the logs from vdsm.log to the metrics store

Comment 2 Shirly Radco 2017-10-16 09:43:09 UTC
Please provide the use cases for using this log.

Comment 3 Yaniv Lavi 2017-10-29 17:04:21 UTC
(In reply to Shirly Radco from comment #2)
> Please provide the use cases for using this log.

Debugging engine-host issues in a single view.
Exploring VMs and tasks bottlenecks on the hosts.

Comment 4 Shirly Radco 2017-11-09 12:58:41 UTC
configure_host_logs_processing is set to false by default.

In order to test user can run
/usr/share/ovirt-engine-metrics/setup/ansible/configure_ovirt_machines_for_metrics.sh -e "configure_host_logs_processing=true"

or add 

configure_host_logs_processing: true

to the /etc/ovirt-engine-metrics/config.yml file.

When updating back to configure_host_logs_processing=false the configuration file is removed from /etc/fluentd/config.d/ and collectd is restarted.

Comment 5 Lukas Svaty 2017-11-09 13:04:35 UTC
Hi Shirly, 

I do not agree with the decision of having it off by default, I see an inconsistency here between 2 collection utilities, log-collector and metrics store.

Log-collector is gathering logs from hosts by default while metrics store is not. 

Where did the decision to have it disabled by default come from? Can we change it to be enabled by default?

LS

Comment 6 Yaniv Lavi 2017-11-09 13:11:47 UTC
Since we see that system resources is not affected, let's move to be on by default.

Comment 7 Shirly Radco 2017-11-13 12:36:19 UTC
I have updated the default value to true.
vdsm.log will now be collected by default.

Comment 8 Shirly Radco 2017-11-15 10:49:12 UTC
Yaniv L requested this to be backport also to 4.1.

Comment 9 Lukas Svaty 2017-11-29 11:31:08 UTC
Currently index for logs is not being created in ViaQ setup, thus this is blocking this verification.

More over I do not see any parsing information of vdsmd.log so it seems to me like FailedQA. Shirly any advise on verification steps here?

For engine log:
Nov 29 11:55:44 example.com fluentd[15159]: 2017-11-29 11:55:44 +0100 [info]: following tail of /var/log/ovirt-engine/engine.log

Nothing for vdsmd.log

Comment 10 Lukas Svaty 2017-11-30 14:20:15 UTC
ignore the last comment, ofcourse you cant follow both engine.log and vdsm.log on a single machine as they are on different machines (host vs engine)

verified in ovirt-engine-metrics-1.0.8-1.el7ev.noarch

Comment 11 Yaniv Lavi 2017-12-03 13:24:22 UTC
*** Bug 1388611 has been marked as a duplicate of this bug. ***