Bug 2017756

Summary:	overlaySize setting on containerruntimeconfig is ignored due to cri-o defaults
Product:	OpenShift Container Platform	Reporter:	Pablo Alonso Rodriguez <palonsor>
Component:	Node	Assignee:	Peter Hunt <pehunt>
Node sub component:	CRI-O	QA Contact:	MinLi <minmli>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	aos-bugs, minmli
Version:	4.7	Keywords:	Reopened
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:22:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Pablo Alonso Rodriguez 2021-10-27 11:48:10 UTC

Description of problem:

If I follow [1] to create a ContainerRuntimeConfig that sets an overlaySize, that setting is ignored and container root filesystems can grow higher than overlaySize.

Version-Release number of selected component (if applicable):

Confirmed on 4.7.31 and latest nightly, likely to occur in any version after the commit I'll post in the additional notes.

How reproducible:

Always in the right versions

Steps to Reproduce:
1. Follow instructions on [1]
2.
3.

Actual results:

Container root filesystem sizes not limited to overlaySize

Expected results:

Container root filesystem sizes limited to overlaySize

Additional info:

It seems that this commit[2] introduced `storage_driver` and `storage_options` settings on crio config, in order to copy them from defaults.

The problem is that, by doing so, all the storage options from /etc/containers/storage.conf are overridden. This includes the "size" setting under "storage.options" section of "/etc/containers/storage.conf" that is configured whenever an overlaySize is specified[3].

Versions prior to that commit (e.g. 4.7.19) don't reproduce the bug. If I also tune cri-o config to remove those options, the issue also doesn't happen.

As a last note, the wrong commit was introduced to have cri-o defaults managed by MCO in a safe way. Some recent versions also ship an /etc/crio/crio.conf that includes the same options, overriding /etc/containers/storage.conf in the very same way.

[1] - https://docs.openshift.com/container-platform/4.9/post_installation_configuration/machine-configuration-tasks.html#set-the-default-max-container-root-partition-size-for-overlay-with-crio_post-install-machine-configuration-tasks
[2] - https://github.com/openshift/machine-config-operator/commit/d6809a18ab361da4b8985a15ae752379bb70bf7e
[3] - https://github.com/openshift/machine-config-operator/blob/d6809a18ab361da4b8985a15ae752379bb70bf7e/pkg/controller/container-runtime-config/helpers.go#L279

Comment 1 Pablo Alonso Rodriguez 2021-10-27 12:41:50 UTC

I coded PR[1] to get this fixed. It basically does 2 things:
- Removes crio options from /etc/crio/crio.conf.d/00-default that collide with /etc/containers/storage.conf from both default master and worker MCP default machineconfigs
- It blanks /etc/crio/crio.conf. I understand this may sound as a quite radical approach, but recent cri-o versions also include the offending settings on it. I already checked that /etc/crio/crio.conf.d/00-default contains any default value needed from /etc/crio/crio.conf , so the whole file is redundant. In fact, as per the commit that introduced the issue[2], this seems to be the original plan.

[1] - https://github.com/openshift/machine-config-operator/pull/2811
[2] - https://github.com/openshift/machine-config-operator/commit/d6809a18ab361da4b8985a15ae752379bb70bf7e

Comment 3 Peter Hunt 2021-10-27 15:22:13 UTC

I believe this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2012838

Comment 4 Pablo Alonso Rodriguez 2021-10-27 15:30:36 UTC

Indeed it is. Thanks and sorry.

*** This bug has been marked as a duplicate of bug 2012838 ***

Comment 5 Pablo Alonso Rodriguez 2021-10-28 16:03:44 UTC

Reopening bug because, although it is technically a duplicate, the suggested set of changes still makes sense alone (it is good to not have duplicated settings).

Check this PR comments for more context[1][2]

[1] - https://github.com/openshift/machine-config-operator/pull/2811#issuecomment-953047861
[2] - https://github.com/openshift/machine-config-operator/pull/2811#issuecomment-953857563

Comment 7 MinLi 2021-12-01 09:08:53 UTC

verified on 4.10.0-0.nightly-2021-12-01-032405

sh-4.4# chroot /host
sh-4.4# cat /etc/crio/crio.conf
# The CRI-O configuration file specifies all of the available configuration
# options and command-line flags for the crio(8) OCI Kubernetes Container Runtime
# daemon, but in a TOML format that can be more easily modified and versioned.
#
# Please refer to crio.conf(5) for details of all configuration options.

# CRI-O supports partial configuration reload during runtime, which can be
# done by sending SIGHUP to the running process. Currently supported options
# are explicitly mentioned with: 'This option supports live configuration
# reload'.

# CRI-O reads its storage defaults from the containers-storage.conf(5) file
# located at /etc/containers/storage.conf. Modify this storage configuration if
# you want to change the system's defaults. If you want to modify storage just
# for CRI-O, you can change the storage configuration options here.
[crio]

# The crio.runtime table contains settings pertaining to the OCI runtime used
# and options for how to set up and manage the OCI runtime.
[crio.runtime]

# If true, SELinux will be used for pod separation on the host.
selinux = true
...

sh-4.4# cat /etc/crio/crio.conf.d/00-default
[crio]
internal_wipe = true
version_file_persist = "/var/lib/crio/version"

[crio.api]
stream_address = ""
stream_port = "10010"

Comment 10 errata-xmlrpc 2022-03-10 16:22:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056