Bug 1573680

Summary:	[RFE] Fluentd handling of long log lines (> 16KB) split by Docker and indexed into several ES documents
Product:	OpenShift Container Platform	Reporter:	Greg Rodriguez II <grodrigu>
Component:	RFE	Assignee:	Jeff Cantrill <jcantril>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Xiaoli Tian <xtian>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.6.0	CC:	abedwards, aos-bugs, dcaldwel, jokerman, jolee, knakayam, mmccomas, mpatel, rkant, rmeggins, scuppett, stwalter, tibrahim, tkatarki
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-03-07 17:55:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Greg Rodriguez II 2018-05-01 23:33:47 UTC

Bug 1422008 previously closed without resolution.  Customer reporting issue persists.  Requesting new RFE.


Description of problem:
Long lines read by fluentd from the Docker logs are split into several documents sent to Elasticsearch.
The max size of the message seems to be 16KB therefore for a message of 85KB the result is that 6 messages were created in different chunks.
Fluentd is configured with the default configuration (docker json-file log driver).

Version-Release number of selected component (if applicable):
OCP v3.3.1.7 

How reproducible:
100%

Steps to Reproduce:
1. oc debug dc/cakephp
2. generate a file with all the content (attached) in a single line.
3. cat longlog.txt

Actual results:
The message is split into 6 messages visible from Kibana

Expected results:
A single message should have been generated

Additional info:
* I have tried to put the document into Elasticsearch manually and it is not split
* oc logs don't show anything
* fluentd logs don't show anything
* docker logs show the entire message

Previous RFE 1422008 closed without resolution - https://bugzilla.redhat.com/show_bug.cgi?id=1422008

Comment 4 Rich Megginson 2018-06-22 14:55:28 UTC

The original issue with docker was that it was running OOM when logging because there was no upper limit on the size of a log entry: https://github.com/moby/moby/issues/18057 so a hard coded limit of 16k was used.

There were various proposals to make the size configurable: https://github.com/moby/moby/issues/34855 and https://github.com/moby/moby/issues/32923#issuecomment-299334898 which were rejected by docker/moby upstream.

We might be able to use https://github.com/fluent-plugins-nursery/fluent-plugin-concat to join split records into a single record.

The docker/moby team also suggest that we write our own plugin that would allow a much higher limit.

Comment 15 Jeff Cantrill 2019-03-07 17:55:29 UTC

Fixed for CRIO use in 3.11 in https://bugzilla.redhat.com/show_bug.cgi?id=1552304.  Closing CURRENTRELEASE with no intention to resolve specifically for docker

Comment 16 Red Hat Bugzilla 2023-09-15 00:07:55 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days