Bug 1458735

This most probably won't be fixed ever, for some critical exceptions we just need stacktrace in INFO, you need to by pass that inside common logging (for example adding whole line to previous message if line doesn't start with "<TIMESTAMP> <LOG LEVEL>"

I updated the fluentd to be able to consume multiline log messages.

ovirt-engine-metrics-1.0.4.2-1.el7ev.noarch

At the moment we throw away the traceback and just logs the error, as mentioned by Martin for components such as mom it is vital to store the traceback to be able to debug issues properly.


See event stored:
EVENT_ID: VDS_FAILURE(12), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout

No traceback afterwards in the logged sample, next log message continues

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

I fixed this in fluentd config, please test.

waiting for new build

Tracebacks available in ovirt-engine-metrics-1.0.4.3-1.el7ev.noarch


{
  "_index": "ovirt-metrics-2017.06.26",
  "_type": "fluentd",
  "_id": "AVzkoJrcGiB38QjWeuEp",
  "_score": null,
  "_source": {
    "level": "ERROR",
    "message": "EVENT_ID: VDS_FAILURE(12), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout\n\tat org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:158)\n\tat org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:121)\n\tat org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73)\n\tat org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)\n\tat org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)\n\tat org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:408)\n\tat org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:677)\n\tat org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:117)\n\tat org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:84)\n\tat org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:275)\n\tat sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:83)\n\tat org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:55)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:213)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.rmi.ConnectException: Connection timeout\n\tat org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.lambda$connect$1(ReactorClient.java:120)\n\tat org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:25)\n\tat org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)\n, Custom Event ID: -1, Message: Host moonmoon is non responsive.",
    "service": "ovirt.engine",
    "tag": "project.ovirt-logs-engine",
    "hostname": "ls-engine1.example.com",
    "ovirt": {
      "class": "org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector",
      "thread": "org.ovirt.thread.pool-7-thread-1",
      "correlationid": "75bf338b"
    },
    "@timestamp": "2017-06-26T15:38:12+02:00"
  },
  "fields": {
    "@timestamp": [
      1498484292000
    ]
  },
  "highlight": {
    "message": [
      "EVENT_ID: VDS_FAILURE(12), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: @kibana-highlighted-field.ConnectException@/kibana-highlighted-field@: Connection timeout\n\tat org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:158)\n\tat org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:121)\n\tat org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73)\n\tat org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)\n\tat org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)\n\tat org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:408)\n\tat org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:677)\n\tat org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:117)\n\tat org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:84)\n\tat org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:275)\n\tat sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:83)\n\tat org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:55)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:213)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: @kibana-highlighted-field.ConnectException@/kibana-highlighted-field@: Connection timeout\n\tat org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.lambda$connect$1(ReactorClient.java:120)\n\tat org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:25)\n\tat org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)\n, Custom Event ID: -1, Message: Host moonmoon is non responsive."
    ]
  },
  "sort": [
    1498484292000
  ]
}