Bug 1573680 - [RFE] Fluentd handling of long log lines (> 16KB) split by Docker and indexed into several ES documents [NEEDINFO]
Summary: [RFE] Fluentd handling of long log lines (> 16KB) split by Docker and indexed...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RFE
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.11.z
Assignee: Jeff Cantrill
QA Contact: Xiaoli Tian
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-01 23:33 UTC by Greg Rodriguez II
Modified: 2019-07-29 07:52 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-07 17:55:29 UTC
Target Upstream Version:
jcantril: needinfo? (tkatarki)


Attachments (Terms of Use)

Description Greg Rodriguez II 2018-05-01 23:33:47 UTC
Bug 1422008 previously closed without resolution.  Customer reporting issue persists.  Requesting new RFE.


Description of problem:
Long lines read by fluentd from the Docker logs are split into several documents sent to Elasticsearch.
The max size of the message seems to be 16KB therefore for a message of 85KB the result is that 6 messages were created in different chunks.
Fluentd is configured with the default configuration (docker json-file log driver).

Version-Release number of selected component (if applicable):
OCP v3.3.1.7 

How reproducible:
100%

Steps to Reproduce:
1. oc debug dc/cakephp
2. generate a file with all the content (attached) in a single line.
3. cat longlog.txt

Actual results:
The message is split into 6 messages visible from Kibana

Expected results:
A single message should have been generated

Additional info:
* I have tried to put the document into Elasticsearch manually and it is not split
* oc logs don't show anything
* fluentd logs don't show anything
* docker logs show the entire message

Previous RFE 1422008 closed without resolution - https://bugzilla.redhat.com/show_bug.cgi?id=1422008

Comment 4 Rich Megginson 2018-06-22 14:55:28 UTC
The original issue with docker was that it was running OOM when logging because there was no upper limit on the size of a log entry: https://github.com/moby/moby/issues/18057 so a hard coded limit of 16k was used.

There were various proposals to make the size configurable: https://github.com/moby/moby/issues/34855 and https://github.com/moby/moby/issues/32923#issuecomment-299334898 which were rejected by docker/moby upstream.

We might be able to use https://github.com/fluent-plugins-nursery/fluent-plugin-concat to join split records into a single record.

The docker/moby team also suggest that we write our own plugin that would allow a much higher limit.

Comment 15 Jeff Cantrill 2019-03-07 17:55:29 UTC
Fixed for CRIO use in 3.11 in https://bugzilla.redhat.com/show_bug.cgi?id=1552304.  Closing CURRENTRELEASE with no intention to resolve specifically for docker


Note You need to log in before you can comment on or make changes to this bug.