Bug 1903983

Marius,

The error indicates a authentication issue.  Did you first login to registry.access.redhat.com with `podman login` prior to doing the pull?

Does this command work for you?  If so, that's a further indication of an authentication error.

podman run -d --name test-sh docker.io/library/alpine sh

Tom - no, this one is actually a known issue with rootless, though we originally started picking it up on the gating test runs for v2.2.0, not 2.1.1. I'm working on resolving it in https://github.com/containers/podman/pull/8561

Can confirm it doesn't work in latest CentOS Stream either:

```
$ podman start mosquitto
Error: unable to start container "6ba0cc3844e1219b515a08a1219993160ae15d0f8c579621622df59e5e0c8887": container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "sysfs" to rootfs at "/sys" caused: operation not permitted: OCI runtime permission denied error

$ rpm -qa | grep -i podman-2.1.1
podman-2.1.1-3.module_el8.4.0+575+63b40ad7.x86_64

$ podman --version
podman version 2.1.1
```

I can also confirm (CentOS Stream 8) with podman 2.1.1.

Fix here is committed and will be in Podman 3.0, landing in 8.4.0. Moving to post. Tagging in Jindrich for errata and packaging.

Created attachment 1756155 [details]
podman-info

The issue persists for me as well:

$ runc --version
runc version spec: 1.0.2-dev

$ rpm -qf /usr/bin/runc
runc-1.0.0-70.rc92.module+el8.4.0+9804+5385893b.x86_64

podman-3.0.0-0.38rc2.module+el8.4.0+9804+5385893b.src.rpm

$ podman run -d --name swamped-crate-system --runtime /usr/bin/runc busybox:latest sleep 1000
Error: time="2021-02-10T03:59:09-05:00" level=error msg="container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"sysfs\" to rootfs at \"/sys\" caused: operation not permitted": OCI permission denied

thanks for confirming it.

I am still not able to reproduce locally, do you have any file under ~/.config/containers?

Could you please attach the output you get when you run?

$ podman --runtime /usr/bin/runc --log-level debug run -d --name swamped-crate-system busybox:latest sleep 1000

@Guiseppe, I don't have any file under ~/.config/containers or any custom configuration in general. Please find the debug run in the attachments.

Created attachment 1756161 [details]
podman-run-debug

thanks that helped.

I think it is a regression in the kernel, I can reproduce on 4.18.0-283

Simpler reproducer:

$ unshare -rmn mount -t sysfs sysfs /sys && echo it works
mount: /sys: permission denied.

expected result (tested both on Fedora 5.10.10-200.fc33 and 4.18.0-240.10.1.el8_3):

$ unshare -rmn mount -t sysfs sysfs /sys && echo it works
it works

The kernel refuses to mount sysfs even if the user namespace owns the network namespace.

Moving to the kernel for further triage.

(In reply to Giuseppe Scrivano from comment #20)
> thanks that helped.
> 
> I think it is a regression in the kernel, I can reproduce on 4.18.0-283
> 
> Simpler reproducer:
> 
> $ unshare -rmn mount -t sysfs sysfs /sys && echo it works
> mount: /sys: permission denied.
> 
> expected result (tested both on Fedora 5.10.10-200.fc33 and
> 4.18.0-240.10.1.el8_3):
> 
> $ unshare -rmn mount -t sysfs sysfs /sys && echo it works
> it works
> 
> The kernel refuses to mount sysfs even if the user namespace owns the
> network namespace.
> 
> Moving to the kernel for further triage.

Did you have a previous mount of sysfs visible in the mount namespace you were mounting sysfs in?

If not it won't work by design.

Hello. Just stumbled across this issue while doing a Google search. I am seeing this on a fresh install of CentOS Stream, fully updated.

Alexey the /sys file system allows containers to look at things like how much memory do they have in their cgroup.

(In reply to Daniel Walsh from comment #31)
> Alexey the /sys file system allows containers to look at things like how
> much memory do they have in their cgroup.

sysfs itself does not provide information about available memory. You are using cgroup2 mounted in a subdirectory.

mount -t tmpfs tmpfs /sys
mkdir -p /sys/fs/cgroup
mount -t cgroup2 cgroup2 /sys/fs/cgroup

Can you just do this ?

I ask because sysfs is more about hardware and loaded kernel modules. I'm not very clear why this information is needed in a container in rootless mode.

Alexey the bottom line is /sys is considered part of the Linux environment and lots of container applications will fail to work if the /sys file system is not seen within the container.

*** Bug 1931800 has been marked as a duplicate of this bug. ***

*** Bug 1732957 has been marked as a duplicate of this bug. ***

List of commits available on kernel-4.18.0-293.el8:
Related commit: 64153f716607 ("Unbreak mount_capable()")

*** Bug 1935609 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1578

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days