Created attachment 1324364 [details] fluent event file from OST Description of problem: Fluentd should process the engine log and forward them as events, but atm every line produces a warning. Actual results: 2017-09-08T12:10:08-04:00 fluent.warn {"message":"got incomplete line before first line from /var/log/ovirt-engine/engine.log: \"2017-09-08 12:10:08,841-04 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] START, GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host-0, VdsIdVDSCommandParametersBase:{hostId='30fba1ba-62eb-4f60-ac27-871c6a0b785d'}), log id: 31ac440c\\n\""} Expected results: A normal event with the log record, for every log level, and exception as well Additional info: Engine.log was in debug mode.
I've checked ovirt's engine log layout and I don't see something special. Simply a line break (\n essentially) at the end of each message, nothing more.
An example of INFO message of the engine 2017-09-08T12:10:34-04:00 fluent.warn {"message":"got incomplete line before first line from /var/log/ovirt-engine/engine.log: \"2017-09-08 12:10:34,810-04 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 44) [] The file '/etc/ovirt-engine/engine.conf' doesn't exist or isn't readable. Will return an empty set of properties.\\n\""}
I checked this and the problem seems to be in the line before the log meddage that is stated in the fluentd.warn. I would like to see the actual engine.log as well as the /var/log/messages so I can see the issue in the engine.log that causes it. Can you please attach fresh /var/log/messages and engine.log?
or fluent event file, since I see you previously attached it.
Every OST job has the engine.log in exported-artifacts. Here is an engine.log of the latest successful for example http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/45/artifact/exported-artifacts/test_logs/hc-basic-suite-master/post-004_basic_sanity.py/lago-hc-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
OST is set to utc-3 timezone. The bug was that the regex expression only accepted 'utc+' and not 'utc-'.
verified in ovirt-engine-metrics-1.0.8-1.el7ev.noarch