Description of problem: log_size_max in crio.conf is not honored. Pod logs always wrap around 12-13MB Version-Release number of selected component (if applicable): CRI-O 1.11.7 on OCP 3.11.31 How reproducible: Always Steps to Reproduce: 1. Edit /etc/crio/crio.conf and set log_size_max = 200000000 (200MB) 2. systemctl restart crio (also tried rebooting 3. Run containers that create a lot of stdout and watch the size of the pods logs under /var/log/pods/<pod id> - can find the symlink under /var/log/containers Actual results: Pod logs will wrap when they reach 12-13MB Expected results: pod logs do not wrap until they reach log_size_max bytes
Could you please share your crio.conf? Did you only tweak log_size_max from the default? Also could you show me the output of "pgrep -fa conmon" while the container is running?
Created attachment 1497185 [details] crio.conf
Created attachment 1497186 [details] pgrep output The container that is putting out lots of logs is centos-logtest. I verified immediately after running pgrep that it was exhibiting the behavior described in this bz - wrapping the log ever 12MB or so.
I forgot to mention, log_size_max was the only change in crio.conf.
thanks for all the extra information. So it is the kubelet that does the log rotation. If you want to increase the size, from the default 10Mi, you should add something like: container-log-max-size: - 200Mi to your /etc/origin/node/node-config.yaml file then restart atomic-openshift-node.service.
Assigning to myself for verification of comment 5 and to ensure docs cover this. Thanks for the info
@gscrivan Are there any other crio.conf flags/params that are overridden by the kubelet? I'd like to get them documented. I found no mention of container-log-max-size in OpenShift or Kube docs.
@Mike, no, there are no other params that need to be overriden.