Bug 1699326

To confirm, you're running as root?

Please add `--log-level=debug` to your options and provide output of the Podman command, plus any entries in syslog from `conmon`.

Also, you may get a better error message if you drop the `-t` flag (`-i` instead of `-it`).

(In reply to Matthew Heon from comment #1)
> To confirm, you're running as root?

Hi Matthew,

no I'm trying to run podman as normal user.
Will attach debug output.

Created attachment 1554788 [details]
podman run --rm --log-level=debug -i fedora:29 echo "Hello world!"

Ahh, it's the 'keycreate' failure. We saw this one when runc updated somewhat recently. I'll check around and see what the resolution was - I believe it might be a new container-selinux package? Thanks for the debug logs.

runc-1.0.0-85.dev.gitdd22a84.fc30 should have a fix - can you check the version of your runc package? It looks like it was pushed to stable two days ago, so it should be available by 'dnf upgrade'

rpm -qa |grep runc
runc-1.0.0-85.dev.gitdd22a84.fc30.x86_64

already installed. I ran dnf update again before writing the report.
Looking at the log I also see
time="2019-04-12T15:42:56+02:00" level=warning msg="Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied"
Not sure if that is a problem.

Are you doing this on a disabled SELinux system?

(In reply to Daniel Walsh from comment #7)
> Are you doing this on a disabled SELinux system?

Yes, indeed.

Well that makes me cry
http://stopdisablingselinux.com/

But this looks like a bug, but I don't think it has much to dow with the conmon issue.

Does the command work if you run it as root.

(In reply to Daniel Walsh from comment #9)
> Well that makes me cry
> http://stopdisablingselinux.com/

Dan, don't cry. I'll explain what happened to you outside of bugzilla.
Maybe you can help me fix it.

> But this looks like a bug, but I don't think it has much to dow with the
> conmon issue.
> 
> Does the command work if you run it as root.

No, even as root it fails. Trace looks similar and ends with
ERRO[0000] container create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"write /proc/self/attr/keycreate: invalid argument\"" : internal libpod error

No problem, send email with your issue, but you might have found a bigger issue.  Since some people sadly continue to disable SELinux.

I will try this in a VM.

After Dan taught me how to re-enable selinux things started looking better.

podman run --rm -i fedora:29 echo "Hello world!"
Hello world!

ok i understand that we shouldn't have disable selinux. but most people disable it on his workstation. and everybody assume this makes the system more usable and more permissive. does it really true that if i disable selinux then i got a more restrictive system???

Now adays, I see very little reason to disable it on the work station. In this case their is a bug in runc that we are racing to fix.

https://github.com/opencontainers/runc/pull/2043

So that runc will stop complaining on SELinux disabled systems.  But I would recommend that you try SELinux on your workstation.


Change 

SELINUX=enforcing
to
SELINUX=permissive

in /etc/selinux/config

touch /.autorelabel
reboot

Should bring the machine up in permissive mode after fixing the labels.  Then you can change the config file back to enforcing
and setenforce 1

And see how everything works.

(In reply to Daniel Walsh from comment #14)
> Now adays, I see very little reason to disable it on the work station. In
> this case their is a bug in runc that we are racing to fix.


I do know how to enable selinux!
I used to enable it on all of our servers (we manage a few thousands).

The real question why I MUST turn it on my workstation? Why any package, system or any program depend on selinux?

Anyway can you tell me ANY good reason why is it good to me to turn it on on my workstation? When it's running on an intranet behind the firewall with a lots of program service and directory structure mounted to different places of my filesystem, many of them encrypted which can't be labeled correctly do not mounted when selinux policy updated etc.? 
Do you really think it will help my daily job???

Well I would say it would protect you system from potentially hostile containers better then any other tool.

The problem is a new version of runc was released that is broken on SELinux disabled machines.  Bottom line is it is trying to do SELinux stuff
even when SELinux is disabled, and this is causing it to break.

If you revert your version of runc to the previous version, it should fix the issue.  I am hoping within the next hour to get a merge into runc
that will fix the SELinux issue and will push out new runc versions as soon as it is merged.

runc-1.0.0-92.dev.gitc1b8c57.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc70b381ad

runc-1.0.0-92.dev.gitc1b8c57.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-6174b47003

(In reply to Daniel Walsh from comment #16)
> Well I would say it would protect you system from potentially hostile
> containers better then any other tool.
> 
> The problem is a new version of runc was released that is broken on SELinux
> disabled machines.  Bottom line is it is trying to do SELinux stuff
> even when SELinux is disabled, and this is causing it to break.
> 
> If you revert your version of runc to the previous version, it should fix
> the issue.  I am hoping within the next hour to get a merge into runc
> that will fix the SELinux issue and will push out new runc versions as soon
> as it is merged.

it's a much better explanation, than it's a bug in container-selinux:-)

but still not convince me to turn on selinux on my workstation.

anyway runc-1.0.0-92.dev.gitc1b8c57.fc29 do not solve the problem just shit it a little bit:

podman run --log-level=debug --rm --env-file .env -v /srv/vidux/mariadb/db:/var/lib/mysql mariadb:10

DEBU[0000] running conmon: /usr/libexec/podman/conmon    args=[-c f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b -u f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b -r /usr/bin/runc -b /home/lfarkas/.local/share/containers/storage/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata -p /tmp/1000/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/pidfile -l /home/lfarkas/.local/share/containers/storage/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --conmon-pidfile /home/lfarkas/.local/share/containers/storage/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/lfarkas/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog]
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
DEBU[0000] Received container pid: -1                   
DEBU[0000] Cleaning up container f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b" 
DEBU[0000] Cleaning up container f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Storage is already unmounted, skipping...    
DEBU[0000] Storage is already unmounted, skipping...    
ERRO[0000] container create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/1000/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/.containerenv\\\" to rootfs \\\"/srv/lfarkas/containers/storage/overlay/6517bc25779fd81615d0f3fc8f3f666dc9622ef7ef627d0ff7289cad2bde2135/merged\\\" at \\\"/srv/lfarkas/containers/storage/overlay/6517bc25779fd81615d0f3fc8f3f666dc9622ef7ef627d0ff7289cad2bde2135/merged/run/.containerenv\\\" caused \\\"open /srv/lfarkas/containers/storage/overlay/6517bc25779fd81615d0f3fc8f3f666dc9622ef7ef627d0ff7289cad2bde2135/merged/run/.containerenv: operation not permitted\\\"\""
: internal libpod error

Well it did fix the original issue you were reporting. Now you are getting into a deeper issue.

This looks like you are running podman as rootless.  Could you try this command with root and see if it works.

If you are running rootless, could you check the ~/.config/container/storage.conf?

Is this setup to run fuse-overlayfs 

Also is ~/.config/container/libpod.conf setup to use cgroupfs rather then systemd for the cgroup_manager?

You might want to cleanup your storage and config

rm -rf ~/.local/share/containers ~/.config/containers

And then see if the command runs.

runc-1.0.0-92.dev.gitc1b8c57.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc70b381ad

runc-1.0.0-92.dev.gitc1b8c57.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-6174b47003

(In reply to Daniel Walsh from comment #20)
> You might want to cleanup your storage and config
> 
> rm -rf ~/.local/share/containers ~/.config/containers
> 
> And then see if the command runs.

rm -rf ~/.local/share/containers/*
helps.

runc-1.0.0-92.dev.gitc1b8c57.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

runc-1.0.0-92.dev.gitc1b8c57.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.