Bug 1746364

Summary:	applying cgroup configuration for process caused \"mountpoint for devices not found\"": OCI runtime error
Product:	[Fedora] Fedora	Reporter:	Lukas Slebodnik <lslebodn>
Component:	podman	Assignee:	Daniel Walsh <dwalsh>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	31	CC:	awilliam, bbaude, dustymabe, dwalsh, frantisek.kluknavsky, jnovy, lsm5, mheon, santiago
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	AcceptedFreezeException
Fixed In Version:	podman-1.5.1-2.17.dev.gitce64c14.fc31	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-03 21:30:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1644938

Description Lukas Slebodnik 2019-08-28 09:31:42 UTC

Description of problem:
The default cgroup hierarchy is set to unified (cgroups v2) (#1732114).
https://fedoraproject.org/wiki/Changes/CGroupsV2
and thus podman does not work on f31 by default

Version-Release number of selected component (if applicable):
sh$ rpm -q systemd podman containernetworking-plugins containers-common
systemd-243~rc2-1.fc31.x86_64
podman-1.5.1-8.16.dev.git74224d9.fc32.x86_64
containernetworking-plugins-0.8.1-7.1.dev.git485be65.fc32.x86_64
containers-common-0.1.40-0.4.dev.git1e2d6f6.fc32.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
1. dnf install -y podman
2. podman run --rm fedora:30 cat /etc/os-release

Actual results:
Error: container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"mountpoint for devices not found\"": OCI runtime error

Expected results:
NAME=Fedora
VERSION="30 (Container Image)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

Additional info:

Comment 1 Lukas Slebodnik 2019-08-28 09:32:50 UTC

I know there is a workaround with kernel comandline option (systemd.unified_cgroup_hierarchy=0)
But I would like to be notified when I can revert that workaround.

And I hope it will be also fixed in f31 but I used rawhide which usually contains
latest podman from upstream.

Comment 2 Daniel Walsh 2019-08-28 13:26:43 UTC

You need to change default oci runtime to crun in /etc/containers/libpod.conf

I updated podman-1.5.1-2.17.dev.gitce64c14.fc31 yesterday to change the default.

Please try this out and see if it fixes the issue.

And then update the karma

https://bodhi.fedoraproject.org/updates/FEDORA-2019-0a601cf11c

Comment 3 Lukas Slebodnik 2019-08-29 07:17:42 UTC

Sanity checks passed for me.
podman-1.5.2-0.36.dev.git1eb6b27.fc32.x86_64
podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64

But it is already after beta freeze https://fedoraproject.org/wiki/Releases/31/Schedule
So you might consider request an  exception for this BZ.

Comment 4 Daniel Walsh 2019-08-29 10:56:23 UTC

How do I request an exception?

Comment 5 Daniel Walsh 2019-08-29 11:01:01 UTC

Fixed in podman-1.5.2-0.36.dev.git1eb6b27.fc32.x86_64

Comment 6 Dusty Mabe 2019-08-29 14:21:04 UTC

(In reply to Daniel Walsh from comment #4)
> How do I request an exception?

Go to the blocker bugs application and sign in with your FAS ID and fill out the form: https://qa.fedoraproject.org/blockerbugs/

Comment 7 Adam Williamson 2019-08-29 15:07:42 UTC

The bug also needs to be open, or else it won't show up on any of the searches we use.

Comment 8 Dusty Mabe 2019-08-29 15:56:08 UTC

It's worth noting that podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64 fixes another problem for me. With current podman:

```
[vagrant@vanilla-f31 ~]$ GODEBUG=tls13=0 podman run --rm -it registry.fedoraproject.org/fedora:30                                                                                                                                            
Trying to pull registry.fedoraproject.org/fedora:30...                                                                                                                                                                                       
Getting image source signatures                                                                                                                                                                                                              
Copying blob e214a48be34a done                                                                                                                                                                                                               
Copying config 1cabdcebde done                                                                                                                                                                                                               
Writing manifest to image destination                                                                                                                                                                                                        Storing signatures                                                                                                                                                                                                                           
Error: systemd cgroup flag passed, but systemd support for managing cgroups is not available: OCI runtime error
```

With `podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64` this is fixed:

```
[vagrant@vanilla-f31 ~]$ GODEBUG=tls13=0 podman run --rm -it registry.fedoraproject.org/fedora:30
Trying to pull registry.fedoraproject.org/fedora:30...
Getting image source signatures
Copying blob e214a48be34a done
Copying config 1cabdcebde done
Writing manifest to image destination
Storing signatures
[root@0cb217550091 /]# 
```

Comment 9 Matthew Heon 2019-08-29 16:56:32 UTC

@Dan - I'll handle the blocker justification

Comment 10 Fedora Blocker Bugs Application 2019-08-29 17:01:15 UTC

Proposed as a Freeze Exception for 31-beta by Fedora user mheon using the blocker tracking app because:

 Podman will not be prepared for the CGroups V2 feature unless this bug is addressed. Almost all container functionality is broken in the present build.

Comment 11 Fedora Update System 2019-08-29 21:34:31 UTC

FEDORA-2019-0a601cf11c has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0a601cf11c

Comment 12 Adam Williamson 2019-08-29 22:06:23 UTC

+1 FE at least for me, while we kick around the criteria proposal.

Comment 13 Adam Williamson 2019-09-03 20:44:25 UTC

Discussed at today's freeze exception review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2019-09-03/f31-blocker-review.2019-09-03-16.01.html . Accepted as a freeze exception as we consider podman a key component these days and want it to work out of the box if possible.

Comment 14 Fedora Update System 2019-09-03 21:30:40 UTC

podman-1.5.1-2.17.dev.gitce64c14.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.