Bug 1699326 - podman Error: error reading container (probably exited) json message: EOF
Summary: podman Error: error reading container (probably exited) json message: EOF
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: runc
Version: 30
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Jan Chaloupka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-12 12:36 UTC by Stefan Assmann
Modified: 2019-05-03 03:41 UTC (History)
8 users (show)

Fixed In Version: runc-1.0.0-92.dev.gitc1b8c57.fc30 runc-1.0.0-92.dev.gitc1b8c57.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-03 00:58:35 UTC
Type: Bug


Attachments (Terms of Use)
podman run --rm --log-level=debug -i fedora:29 echo "Hello world!" (9.46 KB, text/plain)
2019-04-12 13:44 UTC, Stefan Assmann
no flags Details

Description Stefan Assmann 2019-04-12 12:36:41 UTC
Description of problem:
podman fails to run.
# podman run --rm -it fedora:29 echo "Hello world!"
Trying to pull docker.io/library/fedora:29...Getting image source signatures
Copying blob 01eb078129a0 done
Copying config d09302f77c done
Writing manifest to image destination
Storing signatures
Error: error reading container (probably exited) json message: EOF

Version-Release number of selected component (if applicable):
podman-1.2.0-2.git3bd528e.fc30.x86_64

How reproducible:
always

Steps to Reproduce:
1. podman run --rm -it fedora:29 echo "Hello world!"
2.
3.

Comment 1 Matthew Heon 2019-04-12 13:26:04 UTC
To confirm, you're running as root?

Please add `--log-level=debug` to your options and provide output of the Podman command, plus any entries in syslog from `conmon`.

Also, you may get a better error message if you drop the `-t` flag (`-i` instead of `-it`).

Comment 2 Stefan Assmann 2019-04-12 13:43:27 UTC
(In reply to Matthew Heon from comment #1)
> To confirm, you're running as root?

Hi Matthew,

no I'm trying to run podman as normal user.
Will attach debug output.

Comment 3 Stefan Assmann 2019-04-12 13:44:46 UTC
Created attachment 1554788 [details]
podman run --rm --log-level=debug -i fedora:29 echo "Hello world!"

Comment 4 Matthew Heon 2019-04-12 13:51:27 UTC
Ahh, it's the 'keycreate' failure. We saw this one when runc updated somewhat recently. I'll check around and see what the resolution was - I believe it might be a new container-selinux package? Thanks for the debug logs.

Comment 5 Matthew Heon 2019-04-12 14:01:47 UTC
runc-1.0.0-85.dev.gitdd22a84.fc30 should have a fix - can you check the version of your runc package? It looks like it was pushed to stable two days ago, so it should be available by 'dnf upgrade'

Comment 6 Stefan Assmann 2019-04-12 14:06:36 UTC
rpm -qa |grep runc
runc-1.0.0-85.dev.gitdd22a84.fc30.x86_64

already installed. I ran dnf update again before writing the report.
Looking at the log I also see
time="2019-04-12T15:42:56+02:00" level=warning msg="Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied"
Not sure if that is a problem.

Comment 7 Daniel Walsh 2019-04-12 14:25:13 UTC
Are you doing this on a disabled SELinux system?

Comment 8 Stefan Assmann 2019-04-12 14:35:00 UTC
(In reply to Daniel Walsh from comment #7)
> Are you doing this on a disabled SELinux system?

Yes, indeed.

Comment 9 Daniel Walsh 2019-04-12 16:34:35 UTC
Well that makes me cry
http://stopdisablingselinux.com/

But this looks like a bug, but I don't think it has much to dow with the conmon issue.

Does the command work if you run it as root.

Comment 10 Stefan Assmann 2019-04-12 16:47:59 UTC
(In reply to Daniel Walsh from comment #9)
> Well that makes me cry
> http://stopdisablingselinux.com/

Dan, don't cry. I'll explain what happened to you outside of bugzilla.
Maybe you can help me fix it.

> But this looks like a bug, but I don't think it has much to dow with the
> conmon issue.
> 
> Does the command work if you run it as root.

No, even as root it fails. Trace looks similar and ends with
ERRO[0000] container create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"write /proc/self/attr/keycreate: invalid argument\"" : internal libpod error

Comment 11 Daniel Walsh 2019-04-12 16:54:59 UTC
No problem, send email with your issue, but you might have found a bigger issue.  Since some people sadly continue to disable SELinux.

I will try this in a VM.

Comment 12 Stefan Assmann 2019-04-12 17:53:23 UTC
After Dan taught me how to re-enable selinux things started looking better.

podman run --rm -i fedora:29 echo "Hello world!"
Hello world!

Comment 13 Levente Farkas 2019-04-24 09:13:40 UTC
ok i understand that we shouldn't have disable selinux. but most people disable it on his workstation. and everybody assume this makes the system more usable and more permissive. does it really true that if i disable selinux then i got a more restrictive system???

Comment 14 Daniel Walsh 2019-04-24 13:50:45 UTC
Now adays, I see very little reason to disable it on the work station. In this case their is a bug in runc that we are racing to fix.

https://github.com/opencontainers/runc/pull/2043

So that runc will stop complaining on SELinux disabled systems.  But I would recommend that you try SELinux on your workstation.


Change 

SELINUX=enforcing
to
SELINUX=permissive

in /etc/selinux/config

touch /.autorelabel
reboot

Should bring the machine up in permissive mode after fixing the labels.  Then you can change the config file back to enforcing
and setenforce 1

And see how everything works.

Comment 15 Levente Farkas 2019-04-24 14:16:17 UTC
(In reply to Daniel Walsh from comment #14)
> Now adays, I see very little reason to disable it on the work station. In
> this case their is a bug in runc that we are racing to fix.


I do know how to enable selinux!
I used to enable it on all of our servers (we manage a few thousands).

The real question why I MUST turn it on my workstation? Why any package, system or any program depend on selinux?

Anyway can you tell me ANY good reason why is it good to me to turn it on on my workstation? When it's running on an intranet behind the firewall with a lots of program service and directory structure mounted to different places of my filesystem, many of them encrypted which can't be labeled correctly do not mounted when selinux policy updated etc.? 
Do you really think it will help my daily job???

Comment 16 Daniel Walsh 2019-04-24 15:43:42 UTC
Well I would say it would protect you system from potentially hostile containers better then any other tool.

The problem is a new version of runc was released that is broken on SELinux disabled machines.  Bottom line is it is trying to do SELinux stuff
even when SELinux is disabled, and this is causing it to break.

If you revert your version of runc to the previous version, it should fix the issue.  I am hoping within the next hour to get a merge into runc
that will fix the SELinux issue and will push out new runc versions as soon as it is merged.

Comment 17 Fedora Update System 2019-04-24 15:59:26 UTC
runc-1.0.0-92.dev.gitc1b8c57.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc70b381ad

Comment 18 Fedora Update System 2019-04-24 15:59:32 UTC
runc-1.0.0-92.dev.gitc1b8c57.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-6174b47003

Comment 19 Levente Farkas 2019-04-24 17:45:22 UTC
(In reply to Daniel Walsh from comment #16)
> Well I would say it would protect you system from potentially hostile
> containers better then any other tool.
> 
> The problem is a new version of runc was released that is broken on SELinux
> disabled machines.  Bottom line is it is trying to do SELinux stuff
> even when SELinux is disabled, and this is causing it to break.
> 
> If you revert your version of runc to the previous version, it should fix
> the issue.  I am hoping within the next hour to get a merge into runc
> that will fix the SELinux issue and will push out new runc versions as soon
> as it is merged.

it's a much better explanation, than it's a bug in container-selinux:-)

but still not convince me to turn on selinux on my workstation.

anyway runc-1.0.0-92.dev.gitc1b8c57.fc29 do not solve the problem just shit it a little bit:

podman run --log-level=debug --rm --env-file .env -v /srv/vidux/mariadb/db:/var/lib/mysql mariadb:10

DEBU[0000] running conmon: /usr/libexec/podman/conmon    args=[-c f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b -u f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b -r /usr/bin/runc -b /home/lfarkas/.local/share/containers/storage/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata -p /tmp/1000/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/pidfile -l /home/lfarkas/.local/share/containers/storage/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --conmon-pidfile /home/lfarkas/.local/share/containers/storage/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/lfarkas/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog]
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
DEBU[0000] Received container pid: -1                   
DEBU[0000] Cleaning up container f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b" 
DEBU[0000] Cleaning up container f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Storage is already unmounted, skipping...    
DEBU[0000] Storage is already unmounted, skipping...    
ERRO[0000] container create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/1000/overlay-containers/f9ffb1680dd88c96fc4c58b771ab0a38e148e4ea7aa8ce7831d7e8aaad3bad8b/userdata/.containerenv\\\" to rootfs \\\"/srv/lfarkas/containers/storage/overlay/6517bc25779fd81615d0f3fc8f3f666dc9622ef7ef627d0ff7289cad2bde2135/merged\\\" at \\\"/srv/lfarkas/containers/storage/overlay/6517bc25779fd81615d0f3fc8f3f666dc9622ef7ef627d0ff7289cad2bde2135/merged/run/.containerenv\\\" caused \\\"open /srv/lfarkas/containers/storage/overlay/6517bc25779fd81615d0f3fc8f3f666dc9622ef7ef627d0ff7289cad2bde2135/merged/run/.containerenv: operation not permitted\\\"\""
: internal libpod error

Comment 20 Daniel Walsh 2019-04-24 18:06:52 UTC
Well it did fix the original issue you were reporting. Now you are getting into a deeper issue.

This looks like you are running podman as rootless.  Could you try this command with root and see if it works.

If you are running rootless, could you check the ~/.config/container/storage.conf?

Is this setup to run fuse-overlayfs 

Also is ~/.config/container/libpod.conf setup to use cgroupfs rather then systemd for the cgroup_manager?

You might want to cleanup your storage and config

rm -rf ~/.local/share/containers ~/.config/containers

And then see if the command runs.

Comment 21 Fedora Update System 2019-04-24 20:28:07 UTC
runc-1.0.0-92.dev.gitc1b8c57.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc70b381ad

Comment 22 Fedora Update System 2019-04-25 02:25:55 UTC
runc-1.0.0-92.dev.gitc1b8c57.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-6174b47003

Comment 23 Levente Farkas 2019-04-25 14:36:01 UTC
(In reply to Daniel Walsh from comment #20)
> You might want to cleanup your storage and config
> 
> rm -rf ~/.local/share/containers ~/.config/containers
> 
> And then see if the command runs.

rm -rf ~/.local/share/containers/*
helps.

Comment 24 Fedora Update System 2019-05-03 00:58:35 UTC
runc-1.0.0-92.dev.gitc1b8c57.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 25 Fedora Update System 2019-05-03 03:41:54 UTC
runc-1.0.0-92.dev.gitc1b8c57.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.