Description of problem:
The current upstream fluent-plugin-elasticsearch handles bulk index rejections by retrying them, and also retrying other errors. We might be able to do better in rsyslog:
- identify errors that can be retried, such as bulk index rejections, sleep, then resubmit those records to the rsyslog queue
- identify errors that cannot be retried (e.g. schema violations) and write those records to some sort of "dead letter queue" - a local file (could also be the omelasticsearch errorfile)
We need some way to associate the errors with the records, so we will need also to implement support for a unique record identifier.
- rsyslog assigns a unique id to each record
- omelasticsearch plugin uses this as the _id field
- omelasticsearch plugin assembles a bulk index request from many records and submits
- omelasticsearch processes the response from elasticsearch
- if the error is transient, find the original record with the id in the error response and resubmit it
- if the error is permanent, find the original record with the id in the error repsonse and write it to the error file
the trick is mapping the id in the error response back to the original record. We might need some sort of hash table in omelasticsearch - the key is the _id, and the value is the original message.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
This is a must-have feature for common logging and openshift logging.
merged upstream: https://github.com/rsyslog/rsyslog/commit/6d4635efbb13907bf651b1a6e5a545effe84d9d9
@schituku - https://github.com/openshift/origin-aggregated-logging/pull/1259
This configures rsyslog to retry failed elasticsearch operations, and write any "hard" errors to
Verified and pass with rsyslog-elasticsearch-8.24.0-33.el7.x86_64, atomic-openshift-3.11.0-0.19.0. The rejected logs are sent to Elasiticsearch again.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.