Bug 1565219
| Summary: | Add skipempty to liblognorm json field type for rsyslog mmnormalize | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Rich Megginson <rmeggins> | ||||
| Component: | liblognorm | Assignee: | Noriko Hosoi <nhosoi> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Ondrej Moriš <omoris> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 7.6 | CC: | dapospis, dkopecek, jvymazal, mifiedle, mthacker, nhosoi, nkinder, omoris, pportant, rmeggins, schituku | ||||
| Target Milestone: | rc | Keywords: | Patch, Triaged | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | liblognorm-2.0.2-3.el7 | Doc Type: | Enhancement | ||||
| Doc Text: |
When a JSON format log which contains an empty object was passed to rsyslog, there was no method to eliminate the empty object. This enhancement implements it by adding a parameter skipempty to the json field type which is to be configured in the rulebase used by the mmnormalize plugin.
Configuration example:
[name.rulebase]
version=2
rule=:%.:json:skipempty%
[rsyslog configuration]
action(type="mmnormalize" ruleBase="/path/to/name.rulebase")
input
{"message":"Log message","field0":"","field1":[],"field2":{}}
output
{"message":"Log message"}
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1591493 1855422 (view as bug list) | Environment: | |||||
| Last Closed: | 2018-10-30 10:17:13 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1591493 | ||||||
| Bug Blocks: | 1855422, 1990737 | ||||||
| Attachments: |
|
||||||
More details:
1)
For example - the logs from docker json-file look like this - this is what a line in the file looks like - a JSON string, where the "log" field has as its value a JSON encoded string:
{"log":"{\"field1\": \"value1\", \"field2\": \"value2\", \"message\": \"this is the actual message from the app\" }", "stream": "stdin","time": "2018-05-03T17:43:26.653959-06:00"}
And we want the resulting record to look like this:
{"field1":"value1","field2":"value2","message":"this is the actual message from the app","stream":"stdin","time": "2018-05-03T17:43:26.653959-06:00","original_raw_message":"{ \"field1\": \"value1\", \"field2\": \"value2\", \"message\": \"this is the actual message from the app\"}"}
2)
It works in the case of the MESSAGE field from journal because the imjournal.c code sets the MESSAGE value to be the rawmsg in the record, and by default mmjsonparse.c will parse the rawmsg. But we'll also need it to work in the docker json-file case and in the cri-o log file case.
The docker json-file case looks like this
# parse the entire line as a JSON string
action(type="mmjsonparse" cookie="")
# parse the log field as a JSON string
action(type="mmjsonparse" cookie="log")
cri-o is similar, except that we'll need a custom mmnormalize parser for lines read from cri-o
Diffs which are under reviews. Once approved, I'm going to submit then upstream. https://github.com/rsyslog/rsyslog/compare/master...nhosoi:parse_json_extension4 https://github.com/rsyslog/rsyslog-doc/compare/master...nhosoi:parse_json_extension4 https://github.com/rsyslog/libfastjson/compare/master...nhosoi:json_object_array_del_idx [snippet of rsyslog.conf] action(type="mmjsonparse" cookie="" compact="on") action(type="mmjsonparse" cookie="" variable="$!log" container="$!" alt_variable="original_raw_message" compact="on") Upstream PRs: https://github.com/rsyslog/libfastjson/pull/155 https://github.com/rsyslog/rsyslog/pull/2773 -- to pass the builds/tests, libfastjson/pull/155 needs to be merged. https://github.com/rsyslog/rsyslog-doc/pull/691 -- build is failing with "fatal: Couldn't find remote ref refs/pull/691/merge". asking for help. Created attachment 1456078 [details]
test script to verify the bug
How to Run
# su or sudo bash
# bash ./testParseJson.sh
<snip>
Test starts
Kill rsyslogd
Checking valgrind output /tmp/tmp.R1nVEh5pEz, mmjsonparse plugin outputs /tmp/mmjsonparse.log and /tmp/crio.log ...
No memory leaks from mmjsonparse.c -- success
Expected 20 logs are found in /tmp/mmjsonparse.log -- success
"log" value is parsed into the top level -- success
Expected 20 original_raw_messages are found in /tmp/mmjsonparse.log -- success
Verified compact in /tmp/mmjsonparse.log -- success
Expected 10 logs are found in /tmp/crio.log -- success
"log" value is parsed into the top level -- success
Expected 10 original_raw_messages are found in /tmp/crio.log -- success
Verified compact in /tmp/crio.log -- success
Test Result -- success
Safe to remove /tmp/tmp.Qq3yWjjTs0, /tmp/tmp.R1nVEh5pEz, /tmp/mmjsonparse.log and /tmp/crio.log
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:3135 |
Description of problem: many applications write the `message` field which has a value which is a JSON encoded string, and they want the log collector to parse this value and add those fields to the message as top level fields. For example: MESSAGE={"field1":"value1","field2":"value2","message":"this is the actual message from the app"} The resulting message should have { "field1":"value1", "field2":"value2", "message":"this is the actual message from the app", "original_raw_message":"{\"field1\":\"value1\",\"field2\":\"value2\",\"message\":\"this is the actual message from the app\"}" } However, in 8.24.0, there is no way to do this mmjsonparse only works on the entire original message mmnormalize can parse any field as a JSON value, but it cannot put the fields into the top level message, only in a given container. Version-Release number of selected component (if applicable): 8.24.0 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: