Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
When a JSON format log which contains an empty object was passed to rsyslog, there was no method to eliminate the empty object. This enhancement implements it by adding a parameter skipempty to the json field type which is to be configured in the rulebase used by the mmnormalize plugin.
Configuration example:
[name.rulebase]
version=2
rule=:%.:json:skipempty%
[rsyslog configuration]
action(type="mmnormalize" ruleBase="/path/to/name.rulebase")
input
{"message":"Log message","field0":"","field1":[],"field2":{}}
output
{"message":"Log message"}
Description of problem:
many applications write the `message` field which has a value which is a JSON encoded string, and they want the log collector to parse this value and add those fields to the message as top level fields. For example:
MESSAGE={"field1":"value1","field2":"value2","message":"this is the actual message from the app"}
The resulting message should have
{
"field1":"value1",
"field2":"value2",
"message":"this is the actual message from the app",
"original_raw_message":"{\"field1\":\"value1\",\"field2\":\"value2\",\"message\":\"this is the actual message from the app\"}"
}
However, in 8.24.0, there is no way to do this
mmjsonparse only works on the entire original message
mmnormalize can parse any field as a JSON value, but it cannot put the fields into the top level message, only in a given container.
Version-Release number of selected component (if applicable):
8.24.0
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
More details:
1)
For example - the logs from docker json-file look like this - this is what a line in the file looks like - a JSON string, where the "log" field has as its value a JSON encoded string:
{"log":"{\"field1\": \"value1\", \"field2\": \"value2\", \"message\": \"this is the actual message from the app\" }", "stream": "stdin","time": "2018-05-03T17:43:26.653959-06:00"}
And we want the resulting record to look like this:
{"field1":"value1","field2":"value2","message":"this is the actual message from the app","stream":"stdin","time": "2018-05-03T17:43:26.653959-06:00","original_raw_message":"{ \"field1\": \"value1\", \"field2\": \"value2\", \"message\": \"this is the actual message from the app\"}"}
2)
It works in the case of the MESSAGE field from journal because the imjournal.c code sets the MESSAGE value to be the rawmsg in the record, and by default mmjsonparse.c will parse the rawmsg. But we'll also need it to work in the docker json-file case and in the cri-o log file case.
The docker json-file case looks like this
# parse the entire line as a JSON string
action(type="mmjsonparse" cookie="")
# parse the log field as a JSON string
action(type="mmjsonparse" cookie="log")
cri-o is similar, except that we'll need a custom mmnormalize parser for lines read from cri-o
Created attachment 1456078[details]
test script to verify the bug
How to Run
# su or sudo bash
# bash ./testParseJson.sh
<snip>
Test starts
Kill rsyslogd
Checking valgrind output /tmp/tmp.R1nVEh5pEz, mmjsonparse plugin outputs /tmp/mmjsonparse.log and /tmp/crio.log ...
No memory leaks from mmjsonparse.c -- success
Expected 20 logs are found in /tmp/mmjsonparse.log -- success
"log" value is parsed into the top level -- success
Expected 20 original_raw_messages are found in /tmp/mmjsonparse.log -- success
Verified compact in /tmp/mmjsonparse.log -- success
Expected 10 logs are found in /tmp/crio.log -- success
"log" value is parsed into the top level -- success
Expected 10 original_raw_messages are found in /tmp/crio.log -- success
Verified compact in /tmp/crio.log -- success
Test Result -- success
Safe to remove /tmp/tmp.Qq3yWjjTs0, /tmp/tmp.R1nVEh5pEz, /tmp/mmjsonparse.log and /tmp/crio.log
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2018:3135
Description of problem: many applications write the `message` field which has a value which is a JSON encoded string, and they want the log collector to parse this value and add those fields to the message as top level fields. For example: MESSAGE={"field1":"value1","field2":"value2","message":"this is the actual message from the app"} The resulting message should have { "field1":"value1", "field2":"value2", "message":"this is the actual message from the app", "original_raw_message":"{\"field1\":\"value1\",\"field2\":\"value2\",\"message\":\"this is the actual message from the app\"}" } However, in 8.24.0, there is no way to do this mmjsonparse only works on the entire original message mmnormalize can parse any field as a JSON value, but it cannot put the fields into the top level message, only in a given container. Version-Release number of selected component (if applicable): 8.24.0 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: