Bug 1565219 - Add skipempty to liblognorm json field type for rsyslog mmnormalize
Summary: Add skipempty to liblognorm json field type for rsyslog mmnormalize
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: liblognorm
Version: 7.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Noriko Hosoi
QA Contact: Ondrej Moriš
Depends On: 1591493
Blocks: 1855422
TreeView+ depends on / blocked
Reported: 2018-04-09 16:14 UTC by Rich Megginson
Modified: 2020-07-09 20:11 UTC (History)
11 users (show)

Fixed In Version: liblognorm-2.0.2-3.el7
Doc Type: Enhancement
Doc Text:
When a JSON format log which contains an empty object was passed to rsyslog, there was no method to eliminate the empty object. This enhancement implements it by adding a parameter skipempty to the json field type which is to be configured in the rulebase used by the mmnormalize plugin. Configuration example: [name.rulebase] version=2 rule=:%.:json:skipempty% [rsyslog configuration] action(type="mmnormalize" ruleBase="/path/to/name.rulebase") input {"message":"Log message","field0":"","field1":[],"field2":{}} output {"message":"Log message"}
Clone Of:
: 1591493 1855422 (view as bug list)
Last Closed: 2018-10-30 10:17:13 UTC
Target Upstream Version:

Attachments (Terms of Use)
test script to verify the bug (7.79 KB, application/x-shellscript)
2018-07-02 22:38 UTC, Noriko Hosoi
no flags Details

System ID Private Priority Status Summary Last Updated
Github rsyslog liblognorm pull 305 0 'None' open Add a parameter skipempty to the json field type. 2020-12-01 14:33:43 UTC
Red Hat Product Errata RHEA-2018:3135 0 None None None 2018-10-30 10:18:20 UTC

Description Rich Megginson 2018-04-09 16:14:35 UTC
Description of problem:
many applications write the `message` field which has a value which is a JSON encoded string, and they want the log collector to parse this value and add those fields to the message as top level fields.  For example:

MESSAGE={"field1":"value1","field2":"value2","message":"this is the actual message from the app"}

The resulting message should have
  "message":"this is the actual message from the app",
  "original_raw_message":"{\"field1\":\"value1\",\"field2\":\"value2\",\"message\":\"this is the actual message from the app\"}"

However, in 8.24.0, there is no way to do this

mmjsonparse only works on the entire original message

mmnormalize can parse any field as a JSON value, but it cannot put the fields into the top level message, only in a given container.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 2 Noriko Hosoi 2018-05-04 20:15:08 UTC
More details:

For example - the logs from docker json-file look like this - this is what a line in the file looks like - a JSON string, where the "log" field has as its value a JSON encoded string:

{"log":"{\"field1\": \"value1\", \"field2\": \"value2\", \"message\": \"this is the actual message from the app\" }", "stream": "stdin","time": "2018-05-03T17:43:26.653959-06:00"}

And we want the resulting record to look like this:

{"field1":"value1","field2":"value2","message":"this is the actual message from the app","stream":"stdin","time": "2018-05-03T17:43:26.653959-06:00","original_raw_message":"{ \"field1\": \"value1\", \"field2\": \"value2\", \"message\": \"this is the actual message from the app\"}"}

It works in the case of the MESSAGE field from journal because the imjournal.c code sets the MESSAGE value to be the rawmsg in the record, and by default mmjsonparse.c will parse the rawmsg.  But we'll also need it to work in the docker json-file case and in the cri-o log file case.

The docker json-file case looks like this

# parse the entire line as a JSON string
action(type="mmjsonparse" cookie="")
# parse the log field as a JSON string
action(type="mmjsonparse" cookie="log")

cri-o is similar, except that we'll need a custom mmnormalize parser for lines read from cri-o

Comment 3 Noriko Hosoi 2018-06-11 17:32:47 UTC
Diffs which are under reviews.  Once approved, I'm going to submit then upstream.




[snippet of rsyslog.conf]
  action(type="mmjsonparse" cookie="" compact="on")
  action(type="mmjsonparse" cookie="" variable="$!log" container="$!" alt_variable="original_raw_message" compact="on")

Comment 4 Noriko Hosoi 2018-06-13 17:58:18 UTC
Upstream PRs:

-- to pass the builds/tests, libfastjson/pull/155 needs to be merged.

-- build is failing with "fatal: Couldn't find remote ref refs/pull/691/merge".  asking for help.

Comment 18 Noriko Hosoi 2018-07-02 22:38:33 UTC
Created attachment 1456078 [details]
test script to verify the bug

How to Run
# su or sudo bash
# bash ./testParseJson.sh
Test starts
Kill rsyslogd
Checking valgrind output /tmp/tmp.R1nVEh5pEz, mmjsonparse plugin outputs /tmp/mmjsonparse.log and /tmp/crio.log ...
No memory leaks from mmjsonparse.c -- success
Expected 20 logs are found in /tmp/mmjsonparse.log -- success
"log" value is parsed into the top level  -- success
Expected 20 original_raw_messages are found in /tmp/mmjsonparse.log -- success
Verified compact in /tmp/mmjsonparse.log -- success
Expected 10 logs are found in /tmp/crio.log -- success
"log" value is parsed into the top level  -- success
Expected 10 original_raw_messages are found in /tmp/crio.log -- success
Verified compact in /tmp/crio.log -- success
Test Result -- success
Safe to remove /tmp/tmp.Qq3yWjjTs0, /tmp/tmp.R1nVEh5pEz, /tmp/mmjsonparse.log and /tmp/crio.log

Comment 45 errata-xmlrpc 2018-10-30 10:17:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.