Bug 1294168

Summary: RFE : multi-line logging
Product: OpenShift Container Platform Reporter: Miheer Salunke <misalunk>
Component: RFEAssignee: Rich Megginson <rmeggins>
Status: CLOSED CURRENTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.1.0CC: anli, aos-bugs, azipory, bandrade, erich, erjones, ewolinet, fcami, jcantril, jmalde, jokerman, lmeyer, mark.vinkx, mbarrett, mhernon, mmccomas, mmckinst, myllynen, pdwyer, ricferna, rkant, rmeggins, sgaikwad, simon.gunzenreiner, sjr, sspeiche, stwalter, tkatarki, tmanor, trogers, wsun
Target Milestone: ---   
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-07 21:12:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Miheer Salunke 2015-12-25 07:52:12 UTC
1. Proposed title of this feature request  
Multi-line logging
  
3. What is the nature and description of the request?  
We have tested the 3.1 centralized logging feature and saw that (like in v2) you do not treat multiline logging messages. We would like to have some way of handling multiline messages, and out-of-the-box support for this in the JBOSS image.
  
4. Why does the customer need this? (List the business requirements here)  
In v2, we used rsyslog to store the data in a file. Now you provide an elasticsearch/kibana access, but this does not work properly if you do not support multiline messages. Main use case are java exceptions : if you cannot keep the stacktrace as a whole, your metrics in kibana are useless and when you search on a given exception, you do not have the contaxt
  
5. How would the customer like to achieve this? (List the functional requirements here)  
Instead of using docker-like logging to stdout and treating that line based, you should be able to cope with logging in a multiline capable format (like JSON). The transport/storage/acces is ok, but then the log message is preserved, and not broken up into multiple independent lines.
  
6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.  
Provoke a stacktrace in a JBOSS image and make sure it is logged as a single event. The result in kibana should show the complete sctacktrace, not a single line of it.
  
7. Is there already an existing RFE upstream or in Red Hat Bugzilla?



8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)?  
Openshift Enterprise 3.2
    
10. List any affected packages or components.  
Openshift Enterprise 3.2 / JBOSS EAP Openshift image
  
11. Would the customer be able to assist in testing this functionality if implemented? 
Yes. We can provide workloads where we can validate the behaviour.

Comment 2 Rich Megginson 2016-03-28 18:01:48 UTC
Can the customer change their log4j configuration to send stack traces as a single data blob as "stack_trace" field metadata of the log entry?

Comment 3 Rich Megginson 2016-03-28 18:06:21 UTC
Is this a request for multi-line logging in general, or specifically to log and view Java exceptions correctly?  If it is the latter, please change the bug summary to be "RFE: Java exception logging and viewing in Kibana"

Note that multi-line logging in general is extremely difficult, and will require different fluentd configuration for each type of multi-line log.  For example, parsing and viewing a multi-line Java exception is a lot different than a Python exception.

Comment 4 Luke Meyer 2016-03-28 18:36:32 UTC
There is a corresponding github issue https://github.com/openshift/origin-aggregated-logging/issues/28

Comment 12 Dan McPherson 2016-09-22 21:33:11 UTC
*** Bug 1378592 has been marked as a duplicate of this bug. ***

Comment 15 Jeff Cantrill 2016-10-27 19:14:56 UTC
The 3.0 images include (but is disabled) a log formatter to output log entries as JSON.  An example is provided here: https://github.com/knrc/openshift-examples/tree/json-logging-eap6/json-logging-eap6 .  Additional documentation is the deliverable for: https://trello.com/c/k617Elhi which is to advise applications how to produce log messages.  There is nothing we are able to do from the server side to handle the various permutations of logs that will come through the system.

Comment 25 markv 2018-06-15 08:21:18 UTC
+1

Comment 27 Jeff Cantrill 2019-01-07 21:12:23 UTC
Closing current release as this is supported OOTB in EAP7 with the merge of https://github.com/jboss-container-images/jboss-eap-7-openshift-image/pull/23 which sets ENABLE_JSON_LOGGING=true