Red Hat Bugzilla – Bug 1294168
RFE : multi-line logging
Last modified: 2017-12-12 09:15:25 EST
1. Proposed title of this feature request
3. What is the nature and description of the request?
We have tested the 3.1 centralized logging feature and saw that (like in v2) you do not treat multiline logging messages. We would like to have some way of handling multiline messages, and out-of-the-box support for this in the JBOSS image.
4. Why does the customer need this? (List the business requirements here)
In v2, we used rsyslog to store the data in a file. Now you provide an elasticsearch/kibana access, but this does not work properly if you do not support multiline messages. Main use case are java exceptions : if you cannot keep the stacktrace as a whole, your metrics in kibana are useless and when you search on a given exception, you do not have the contaxt
5. How would the customer like to achieve this? (List the functional requirements here)
Instead of using docker-like logging to stdout and treating that line based, you should be able to cope with logging in a multiline capable format (like JSON). The transport/storage/acces is ok, but then the log message is preserved, and not broken up into multiple independent lines.
6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
Provoke a stacktrace in a JBOSS image and make sure it is logged as a single event. The result in kibana should show the complete sctacktrace, not a single line of it.
7. Is there already an existing RFE upstream or in Red Hat Bugzilla?
8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)?
Openshift Enterprise 3.2
10. List any affected packages or components.
Openshift Enterprise 3.2 / JBOSS EAP Openshift image
11. Would the customer be able to assist in testing this functionality if implemented?
Yes. We can provide workloads where we can validate the behaviour.
Can the customer change their log4j configuration to send stack traces as a single data blob as "stack_trace" field metadata of the log entry?
Is this a request for multi-line logging in general, or specifically to log and view Java exceptions correctly? If it is the latter, please change the bug summary to be "RFE: Java exception logging and viewing in Kibana"
Note that multi-line logging in general is extremely difficult, and will require different fluentd configuration for each type of multi-line log. For example, parsing and viewing a multi-line Java exception is a lot different than a Python exception.
There is a corresponding github issue https://github.com/openshift/origin-aggregated-logging/issues/28
*** Bug 1378592 has been marked as a duplicate of this bug. ***
The 3.0 images include (but is disabled) a log formatter to output log entries as JSON. An example is provided here: https://github.com/knrc/openshift-examples/tree/json-logging-eap6/json-logging-eap6 . Additional documentation is the deliverable for: https://trello.com/c/k617Elhi which is to advise applications how to produce log messages. There is nothing we are able to do from the server side to handle the various permutations of logs that will come through the system.