Bug 1565214

Description of problem:
The current upstream fluent-plugin-elasticsearch handles bulk index rejections by retrying them, and also retrying other errors.  We might be able to do better in rsyslog:

- identify errors that can be retried, such as bulk index rejections, sleep, then resubmit those records to the rsyslog queue
- identify errors that cannot be retried (e.g. schema violations) and write those records to some sort of "dead letter queue" - a local file (could also be the omelasticsearch errorfile)

We need some way to associate the errors with the records, so we will need also to implement support for a unique record identifier.

- rsyslog assigns a unique id to each record
- omelasticsearch plugin uses this as the _id field
- omelasticsearch plugin assembles a bulk index request from many records and submits
- omelasticsearch processes the response from elasticsearch
- if the error is transient, find the original record with the id in the error response and resubmit it
- if the error is permanent, find the original record with the id in the error repsonse and write it to the error file

the trick is mapping the id in the error response back to the original record.  We might need some sort of hash table in omelasticsearch - the key is the _id, and the value is the original message.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

This is a must-have feature for common logging and openshift logging.