Bug 2027926 - sandbox creation fails due to obsolete option in /etc/containers/storage.conf
Summary: sandbox creation fails due to obsolete option in /etc/containers/storage.conf
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.9
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.9.z
Assignee: Peter Hunt
QA Contact: Sunil Choudhary
Depends On: 2027927 2109599
Blocks: 2108699
TreeView+ depends on / blocked
Reported: 2021-12-01 04:19 UTC by Fraser Tweedale
Modified: 2022-07-21 15:43 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2022-01-04 18:41:24 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2848 0 None open [release-4.9] Bug 2027926: storage.conf: remove obsolete option override_kernel_check 2021-12-09 11:50:27 UTC
Red Hat Product Errata RHBA-2021:5214 0 None None None 2022-01-04 18:41:49 UTC

Description Fraser Tweedale 2021-12-01 04:19:07 UTC
Description of problem:

The `storage.conf(5)` `override_kernel_check` option was removed from the
containers/storage library in early 2019:


With recent version of CRI-O present in OCP >= 4.9, the presence of
this field causes sandbox creation failure when using user namespaces:

    Warning  FailedCreatePodSandBox  SSs (xN over MMm)  kubelet
    (combined from similar events): Failed to create pod sandbox: rpc
    error: code = Unknown desc = error creating pod sandbox with name
    "{NAME}": error creating an ID-mapped copy of layer "{HASH}":
    time="{TIMESTAMP}" level=warning msg="Failed to decode the keys
    [\"storage.options.override_kernel_check\"] from

Version-Release number of selected component (if applicable): OCP 4.9.any

How reproducible: always, when requesting user namespace

Steps to Reproduce:

See blog post: https://frasertweedale.github.io/blog-redhat/posts/2021-07-22-openshift-systemd-workload-demo.html

Actual results: sandbox creation fails (see error message above)

Expected results: sandbox creation succeeds

Additional info:

Removal of `override_kernel_check` option from /etc/containers/storage.conf resolves the issue.

MCO PR (already merged): https://github.com/openshift/machine-config-operator/pull/2845

Backport PR: https://github.com/openshift/machine-config-operator/pull/2848

It is possible that CRI-O itself could be be modified to handle this situation.
The offending code comes from containers/storage/store.go (this library is vendored in
the cri-o repo):

  mappedLayer, _, err := rlstore.Put("", parentLayer, nil, layer.MountLabel, nil, &layerOptions, false, nil, nil) 
  if err != nil {
          return nil, errors.Wrapf(err, "error creating an ID-mapped copy of layer %q", layer.ID)                               
In this case, mappedLayer is non-nil, and err is *also* non-nil and represents a warning
rather than an unrecoverable error.  The program could optimistically continue when mappedLayer
is non-nil.

Comment 6 errata-xmlrpc 2022-01-04 18:41:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.12 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.