Bug 2017756 - overlaySize setting on containerruntimeconfig is ignored due to cri-o defaults
Summary: overlaySize setting on containerruntimeconfig is ignored due to cri-o defaults
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Peter Hunt
QA Contact: MinLi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-27 11:48 UTC by Pablo Alonso Rodriguez
Modified: 2022-03-10 16:22 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:22:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2811 0 None open Bug 2017756: Remove crio settings that overwrite /etc/containers/storage.conf 2021-10-28 16:04:12 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:22:25 UTC

Description Pablo Alonso Rodriguez 2021-10-27 11:48:10 UTC
Description of problem:

If I follow [1] to create a ContainerRuntimeConfig that sets an overlaySize, that setting is ignored and container root filesystems can grow higher than overlaySize.

Version-Release number of selected component (if applicable):

Confirmed on 4.7.31 and latest nightly, likely to occur in any version after the commit I'll post in the additional notes.

How reproducible:

Always in the right versions

Steps to Reproduce:
1. Follow instructions on [1]
2.
3.

Actual results:

Container root filesystem sizes not limited to overlaySize

Expected results:

Container root filesystem sizes limited to overlaySize

Additional info:

It seems that this commit[2] introduced `storage_driver` and `storage_options` settings on crio config, in order to copy them from defaults.

The problem is that, by doing so, all the storage options from /etc/containers/storage.conf are overridden. This includes the "size" setting under "storage.options" section of "/etc/containers/storage.conf" that is configured whenever an overlaySize is specified[3].

Versions prior to that commit (e.g. 4.7.19) don't reproduce the bug. If I also tune cri-o config to remove those options, the issue also doesn't happen.

As a last note, the wrong commit was introduced to have cri-o defaults managed by MCO in a safe way. Some recent versions also ship an /etc/crio/crio.conf that includes the same options, overriding /etc/containers/storage.conf in the very same way.

[1] - https://docs.openshift.com/container-platform/4.9/post_installation_configuration/machine-configuration-tasks.html#set-the-default-max-container-root-partition-size-for-overlay-with-crio_post-install-machine-configuration-tasks
[2] - https://github.com/openshift/machine-config-operator/commit/d6809a18ab361da4b8985a15ae752379bb70bf7e
[3] - https://github.com/openshift/machine-config-operator/blob/d6809a18ab361da4b8985a15ae752379bb70bf7e/pkg/controller/container-runtime-config/helpers.go#L279

Comment 1 Pablo Alonso Rodriguez 2021-10-27 12:41:50 UTC
I coded PR[1] to get this fixed. It basically does 2 things:
- Removes crio options from /etc/crio/crio.conf.d/00-default that collide with /etc/containers/storage.conf from both default master and worker MCP default machineconfigs
- It blanks /etc/crio/crio.conf. I understand this may sound as a quite radical approach, but recent cri-o versions also include the offending settings on it. I already checked that /etc/crio/crio.conf.d/00-default contains any default value needed from /etc/crio/crio.conf , so the whole file is redundant. In fact, as per the commit that introduced the issue[2], this seems to be the original plan.

[1] - https://github.com/openshift/machine-config-operator/pull/2811
[2] - https://github.com/openshift/machine-config-operator/commit/d6809a18ab361da4b8985a15ae752379bb70bf7e

Comment 3 Peter Hunt 2021-10-27 15:22:13 UTC
I believe this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2012838

Comment 4 Pablo Alonso Rodriguez 2021-10-27 15:30:36 UTC
Indeed it is. Thanks and sorry.

*** This bug has been marked as a duplicate of bug 2012838 ***

Comment 5 Pablo Alonso Rodriguez 2021-10-28 16:03:44 UTC
Reopening bug because, although it is technically a duplicate, the suggested set of changes still makes sense alone (it is good to not have duplicated settings).

Check this PR comments for more context[1][2]

[1] - https://github.com/openshift/machine-config-operator/pull/2811#issuecomment-953047861
[2] - https://github.com/openshift/machine-config-operator/pull/2811#issuecomment-953857563

Comment 7 MinLi 2021-12-01 09:08:53 UTC
verified on 4.10.0-0.nightly-2021-12-01-032405

sh-4.4# chroot /host
sh-4.4# cat /etc/crio/crio.conf
# The CRI-O configuration file specifies all of the available configuration
# options and command-line flags for the crio(8) OCI Kubernetes Container Runtime
# daemon, but in a TOML format that can be more easily modified and versioned.
#
# Please refer to crio.conf(5) for details of all configuration options.

# CRI-O supports partial configuration reload during runtime, which can be
# done by sending SIGHUP to the running process. Currently supported options
# are explicitly mentioned with: 'This option supports live configuration
# reload'.

# CRI-O reads its storage defaults from the containers-storage.conf(5) file
# located at /etc/containers/storage.conf. Modify this storage configuration if
# you want to change the system's defaults. If you want to modify storage just
# for CRI-O, you can change the storage configuration options here.
[crio]

# The crio.runtime table contains settings pertaining to the OCI runtime used
# and options for how to set up and manage the OCI runtime.
[crio.runtime]

# If true, SELinux will be used for pod separation on the host.
selinux = true
...

sh-4.4# cat /etc/crio/crio.conf.d/00-default
[crio]
internal_wipe = true
version_file_persist = "/var/lib/crio/version"

[crio.api]
stream_address = ""
stream_port = "10010"

Comment 10 errata-xmlrpc 2022-03-10 16:22:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.