Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionWouter Hummelink
2018-11-19 13:31:36 UTC
Description of problem:
When running podman an error occurs starting a rootless container
Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux release 8.0 Beta (Ootpa)
podman.x86_64 0.10.1.3-5.gitdb08685.el8+2131+7e3e9e07
How reproducible:
Steps to Reproduce:
1. dnf install @container-tools:1.0/default
2. mount -o remount,noexec /home
3. podman run centos:7
Actual results:
standard_init_linux.go:203: exec user process caused "permission denied"
Expected results:
running container
Additional info:
There are several warnings before the actual crash. Removing the mounts.conf share for secrets did not help.
DEBU[0095] Start untar layer
DEBU[0099] Untar time: 4.188800822s
DEBU[0099] setting image creation date to 2018-10-09 18:19:48.447478476 +0000 UTC
DEBU[0099] created new image ID "75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0099] set names of image "75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" to [docker.io/library/centos:7]
DEBU[0099] saved image metadata "{}"
DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]docker.io/library/centos:7"
DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0099] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0099] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
WARN[0099] AppArmor security is not available in rootless mode
DEBU[0099] Using bridge netmode
DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0099] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0099] Creating dest directory: /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676
DEBU[0099] Calling TarUntar(/home/vagrant/.local/share/containers/storage/vfs/dir/f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f, /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676)
DEBU[0099] TarUntar(/home/vagrant/.local/share/containers/storage/vfs/dir/f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676)
DEBU[0102] created container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9"
DEBU[0102] container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" has work directory "/home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata"
DEBU[0102] container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" has run directory "/run/user/1000/run/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata"
DEBU[0102] New container created "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9"
DEBU[0102] container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" has CgroupParent "/libpod_parent/libpod-73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9"
DEBU[0102] Not attaching to stdin
DEBU[0102] mounted container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" at "/home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676"
DEBU[0102] Created root filesystem for container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 at /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676
WARN[0102] error mounting secrets, skipping: getting host secret data failed: failed to read secrets from "/usr/share/rhel/secrets": open /usr/share/rhel/secrets: permission denied
DEBU[0102] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0102] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0102] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d"
DEBU[0102] Created OCI spec for container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 at /home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata/config.json
DEBU[0102] /usr/libexec/podman/conmon messages will be logged to syslog
DEBU[0102] running conmon: /usr/libexec/podman/conmon args=[-c 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 -u 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 -r /usr/bin/runc -b /home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata -p /run/user/1000/run/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata/pidfile -l /home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog]
WARN[0102] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied
DEBU[0102] Received container pid: 29445
DEBU[0102] Created container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 in OCI runtime
DEBU[0102] Enabling signal proxying
DEBU[0102] Attaching to container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9
DEBU[0102] connecting to socket /run/user/1000/libpod/tmp/socket/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/attach
DEBU[0102] Started container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9
standard_init_linux.go:203: exec user process caused "permission denied"
DEBU[0102] Checking container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 status...
DEBU[0102] Cleaning up container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9
DEBU[0102] Network is already cleaned up, skipping...
DEBU[0103] unmounted container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9"
DEBU[0103] Successfully cleaned up container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9
I don't think we can fix this on the Podman side - we put rootless containers in the user's home directory, and mounting said directory noexec means we can't run anything in those containers.
It is possible to work around this by manually specifying a container storage path that's not in a noexec mount. Copying /etc/containers/storage.conf to ~/.config/containers/ (creating the directory if necessary), you can adjust the paths so runroot points to a unique directory in /tmp (/run/user/$UID/libpod/storage/ is generally what we use, I think?) and graphroot to a directory that is not on a noexec mount, your user has read/write privileges on, and is not on a tmpfs.
These proposed solutions do make sense. Maybe add something in documentation about these gotchas?
(And possibly make the error message a bit more descriptive about the actual error)
Wouter, I think the error message is the best we can do. We are telling you that exec is denied.
I am not crazy about podman attempting to diagnose the issue. Since there are many reasons to get permission
denied.
We can add some information to the troublehshooting guide on the github. But not sure where we would put
other information which would help out this situation.
Description of problem: When running podman an error occurs starting a rootless container Version-Release number of selected component (if applicable): Red Hat Enterprise Linux release 8.0 Beta (Ootpa) podman.x86_64 0.10.1.3-5.gitdb08685.el8+2131+7e3e9e07 How reproducible: Steps to Reproduce: 1. dnf install @container-tools:1.0/default 2. mount -o remount,noexec /home 3. podman run centos:7 Actual results: standard_init_linux.go:203: exec user process caused "permission denied" Expected results: running container Additional info: There are several warnings before the actual crash. Removing the mounts.conf share for secrets did not help. DEBU[0095] Start untar layer DEBU[0099] Untar time: 4.188800822s DEBU[0099] setting image creation date to 2018-10-09 18:19:48.447478476 +0000 UTC DEBU[0099] created new image ID "75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0099] set names of image "75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" to [docker.io/library/centos:7] DEBU[0099] saved image metadata "{}" DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]docker.io/library/centos:7" DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0099] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0099] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" WARN[0099] AppArmor security is not available in rootless mode DEBU[0099] Using bridge netmode DEBU[0099] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0099] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0099] Creating dest directory: /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676 DEBU[0099] Calling TarUntar(/home/vagrant/.local/share/containers/storage/vfs/dir/f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f, /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676) DEBU[0099] TarUntar(/home/vagrant/.local/share/containers/storage/vfs/dir/f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676) DEBU[0102] created container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" DEBU[0102] container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" has work directory "/home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata" DEBU[0102] container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" has run directory "/run/user/1000/run/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata" DEBU[0102] New container created "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" DEBU[0102] container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" has CgroupParent "/libpod_parent/libpod-73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" DEBU[0102] Not attaching to stdin DEBU[0102] mounted container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" at "/home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676" DEBU[0102] Created root filesystem for container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 at /home/vagrant/.local/share/containers/storage/vfs/dir/c9d3e7aa42a0d175be34b27b4f2c9ade00563554fd982e9da332933ec7982676 WARN[0102] error mounting secrets, skipping: getting host secret data failed: failed to read secrets from "/usr/share/rhel/secrets": open /usr/share/rhel/secrets: permission denied DEBU[0102] /etc/system-fips does not exist on host, not mounting FIPS mode secret DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0102] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0102] exporting opaque data as blob "sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0102] parsed reference into "[vfs@/home/vagrant/.local/share/containers/storage+/run/user/1000/run]@75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d" DEBU[0102] Created OCI spec for container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 at /home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata/config.json DEBU[0102] /usr/libexec/podman/conmon messages will be logged to syslog DEBU[0102] running conmon: /usr/libexec/podman/conmon args=[-c 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 -u 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 -r /usr/bin/runc -b /home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata -p /run/user/1000/run/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata/pidfile -l /home/vagrant/.local/share/containers/storage/vfs-containers/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog] WARN[0102] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied DEBU[0102] Received container pid: 29445 DEBU[0102] Created container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 in OCI runtime DEBU[0102] Enabling signal proxying DEBU[0102] Attaching to container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 DEBU[0102] connecting to socket /run/user/1000/libpod/tmp/socket/73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9/attach DEBU[0102] Started container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 standard_init_linux.go:203: exec user process caused "permission denied" DEBU[0102] Checking container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 status... DEBU[0102] Cleaning up container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9 DEBU[0102] Network is already cleaned up, skipping... DEBU[0103] unmounted container "73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9" DEBU[0103] Successfully cleaned up container 73a4fb9769188ae5dc51cb7e24b9f2752a4af7b802a8949f06a7b2f2363ab0e9