Description of problem: I tried to run simple command with podman but it failed Version-Release number of selected component (if applicable): sh$ rpm -q podman podman-1.1.2-1.git0ad9b6b.fc29.x86_64 How reproducible: Deterministic Steps to Reproduce: 1. dnf install -y podman 2. podman run --rm registry.access.redhat.com/rhel7/ipa-server bash -c 'cat /etc/os-release' Actual results: sh-4.4# podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES sh-4.4# podman images REPOSITORY TAG IMAGE ID CREATED SIZE registry.access.redhat.com/rhel7/ipa-server latest 3d96d276e837 5 weeks ago 712 MB sh-4.4# podman run --rm registry.access.redhat.com/rhel7/ipa-server bash -c 'cat /etc/os-release' Error: container create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/189e5741c55a65e4dd88f9df3ceb7dfe466cc253821ae6d45f82e55df12e6eed/merged\\\" at \\\"/var/log/journal\\\" caused \\\"mkdir /var/lib/containers/storage/overlay/189e5741c55a65e4dd88f9df3ceb7dfe466cc253821ae6d45f82e55df12e6eed/merged/var/log: file exists\\\"\"" : internal libpod error Expected results: sh-4.4# podman run --rm registry.access.redhat.com/rhel7/ipa-server bash -c 'cat /etc/os-release' NAME="Red Hat Enterprise Linux Server" VERSION="7.6 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.6" PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.6 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.6" Additional info: It works like a magic with docker sh-4.4# rpm -q docker docker-1.13.1-65.git1185cfd.fc29.x86_64 sh-4.4# docker run --rm registry.access.redhat.com/rhel7/ipa-server bash -c 'cat /etc/os-release' NAME="Red Hat Enterprise Linux Server" VERSION="7.6 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.6" PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.6 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
This looks like it could be a runc error - it's failing to mount a tmpfs because a directory exists?
sh$ rpm -qf /usr/bin/runc runc-1.0.0-68.dev.git6635b4f.fc29.x86_64
And the latest version in rawhide/koji have the same issue sh# rpm -q runc runc-1.0.0-80.dev.git923a8f8.fc31.x86_64 sh# podman run --rm registry.access.redhat.com/rhel7/ipa-server bash -c 'cat /etc/os-release' Error: container create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"tmpfs\\\" to rootfs \\\"/var/lib/containers/storage/overlay/69e0282a15448a000847f06114af6f4f9ade85fe22dcad4834246d19c6533836/merged\\\" at \\\"/var/log/journal\\\" caused \\\"mkdir /var/lib/containers/storage/overlay/69e0282a15448a000847f06114af6f4f9ade85fe22dcad4834246d19c6533836/merged/var/log: file exists\\\"\"" : internal libpod error
Reproduces on master and 1.0 as well. Error is definitely coming out of runc, and seems specific to this image. Any chance you can point to where the Dockerfile for this image lives?
This looks like a systemd container.
Not a runc error, reproduces identically with crun. Looking into it. May well be our systemd handling.
Found the problem - /var/log/ in the image is a symlink. Runc does not appear to be equipped to handle mounting a tmpfs inside a directory that is a symlink, and we need to mount /var/log/journal as a tmpfs. Docker must do some additional parsing to properly resolve symlinks.
Opened https://github.com/containers/libpod/pull/2593 to fix upstream
(In reply to Matthew Heon from comment #9) > Opened https://github.com/containers/libpod/pull/2593 to fix upstream I can see that patch was merged a week ago. Could you prepare build for fedora ?
Lukas we are waiting for completion of podman healthcheck command, and then we will cut podman 1.3 release. Hopefully within a week or two.
(In reply to Daniel Walsh from comment #11) > Lukas we are waiting for completion of podman healthcheck command, and then > we will cut podman 1.3 release. > Hopefully within a week or two. Could you at least prepare scrach build for me? Even with pre-release of 1.3. I would like to check whether it helps to pass whole test-suite. (it pass with docker so far)
Could you grab the package from rawhide and try it out. It gets built based on each PR merge.
(In reply to Daniel Walsh from comment #13) > Could you grab the package from rawhide and try it out. It gets built based > on each PR merge. fair enough
podman-1.2.0-2.git3bd528e.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-8875ae3cff
podman-1.2.0-2.git3bd528e.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-dede809f2a
podman-1.2.0-2.git3bd528e.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-8875ae3cff
podman-1.2.0-2.git3bd528e.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-dede809f2a
podman-1.2.0-2.git3bd528e.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
podman-1.2.0-2.git3bd528e.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.