Bug 1642547

Summary: CRI-O 1.11.7: log_size_max is ignored. Logs always wrap at 12-13MB
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: ContainersAssignee: Mike Fiedler <mifiedle>
Status: CLOSED NOTABUG QA Contact: Mike Fiedler <mifiedle>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, gscrivan, jokerman, mifiedle, mmccomas, mpatel
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-22 18:48:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1552304    
Attachments:
Description Flags
crio.conf
none
pgrep output none

Description Mike Fiedler 2018-10-24 16:06:47 UTC 
Description of problem:

log_size_max in crio.conf is not honored.  Pod logs always wrap around 12-13MB

Version-Release number of selected component (if applicable): CRI-O 1.11.7 on OCP 3.11.31


How reproducible: Always


Steps to Reproduce:
1. Edit /etc/crio/crio.conf and set log_size_max = 200000000  (200MB)
2. systemctl restart crio (also tried rebooting
3. Run containers that create a lot of stdout and watch the size of the pods logs under /var/log/pods/<pod id>  - can find the symlink under /var/log/containers

Actual results:

Pod logs will wrap when they reach 12-13MB


Expected results:

pod logs do not wrap until they reach log_size_max bytes
Comment 1 Giuseppe Scrivano 2018-10-24 19:03:27 UTC 
Could you please share your crio.conf?  Did you only tweak log_size_max from the default?

Also could you show me the output of "pgrep -fa conmon" while the container is running?
Comment 2 Mike Fiedler 2018-10-24 19:27:09 UTC 
Created attachment 1497185 [details]
crio.conf
Comment 3 Mike Fiedler 2018-10-24 19:29:04 UTC 
Created attachment 1497186 [details]
pgrep output

The container that is putting out lots of logs is centos-logtest.  I verified immediately after running pgrep that it was exhibiting the behavior described in this bz - wrapping the log ever 12MB or so.
Comment 4 Mike Fiedler 2018-10-24 19:30:07 UTC 
I forgot to mention, log_size_max was the only change in crio.conf.
Comment 5 Giuseppe Scrivano 2018-10-24 21:05:22 UTC 
thanks for all the extra information.

So it is the kubelet that does the log rotation.  If you want to increase the size, from the default 10Mi, you should add something like:

  container-log-max-size:
  - 200Mi

to your /etc/origin/node/node-config.yaml file then restart atomic-openshift-node.service.
Comment 6 Mike Fiedler 2018-10-25 01:00:27 UTC 
Assigning to myself for verification of comment 5 and to ensure docs cover this.   Thanks for the info
Comment 7 Mike Fiedler 2018-10-29 11:55:49 UTC 
@gscrivan  Are there any other crio.conf flags/params that are overridden by the kubelet?   I'd like to get them documented.  I found no mention of container-log-max-size in OpenShift or Kube docs.
Comment 9 Giuseppe Scrivano 2018-11-12 12:15:38 UTC 
@Mike, no, there are no other params that need to be overriden.