Bug 2166225
| Summary: | Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Alex Jia <ajia> |
| Component: | buildah | Assignee: | Gabriela Nečasová <gnecasov> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Alex Jia <ajia> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 9.2 | CC: | atomic-bugs, dwalsh, gnecasov, gscrivan, jnovy, lsm5, mbearer, mboddu, pthomas, tsweeney, umohnani, ypu |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | buildah-1.29.0-2.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 2166195 | Environment: | |
| Last Closed: | 2023-03-06 19:18:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2166195, 2178263 | ||
| Bug Blocks: | |||
|
Description
Alex Jia
2023-02-01 08:23:06 UTC
[root@kvm-04-guest14 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 9.2 Beta (Plow) [root@kvm-04-guest14 ~]# rpm -q podman crun systemd kernel podman-4.3.1-3.el9.x86_64 crun-1.7.2-2.el9.x86_64 systemd-252-3.el9.x86_64 kernel-5.14.0-247.el9.x86_64 [root@kvm-04-guest14 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-1 Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-1... Getting image source signatures Copying blob 612007f8c61d done Copying blob 84d5648acfd5 done Copying config 741abe95c7 done Writing manifest to image destination Storing signatures [root@8d95b772061e /]# rpm -q buildah fuse-overlayfs buildah-1.28.0-2.el9.x86_64 fuse-overlayfs-1.10-1.el9.x86_64 [root@8d95b772061e /]# buildah --log-level=debug info DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] overlay: imagestore=/var/lib/shared DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] overlay: imagestore=/var/lib/shared DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs WARN[0000] failed to shutdown storage: "mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted" that seems a regression caused by https://github.com/containers/common/commit/f39f2a3f8c7680b9e456b9d235570e511807d6c6 that drops sys_chroot from the default capabilities. Dan, should we add the capability back or is the expectation to use --cap-add sys_chroot for build containers? Why does a mount need CAP_SYS_CHROOT? This seems to be working fine in Fedora, why is this broken just in RHEL? Ok so this is buildah --isolation chroot within a container. Is this a regression in RHEL9? I don't think so, I thought we defaulted to no CHROOT at the release of RHEL9? the commit is quite recent: Nov 22, 2022, so it might not be in older releases. Should we consider it a breaking change? If not, I think it is safer to not add CAP_SYS_CHROOT by default to all containers, since it is needed in very few cases (like buildah --isolation chroot). Would it be enough to document the requirement of using --cap-add sys_chroot? Yes I think we should improve the documentation. And recommend users do --cap-add sys_chroot. Alex, could you please try adding --cap-add sys_chroot? (In reply to Giuseppe Scrivano from comment #11) > Alex, could you please try adding --cap-add sys_chroot? It works well for me with buildah-1.29.0-2.el9 and buildah-1.29.0-3.el9. [root@kvm-04-guest15 ~]# podman run --cap-add sys_chroot --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-1 ...ignore... [root@ae3bc3f4dc57 /]# rpm -Uvh buildah-1.29.0-3.el9.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:buildah-1:1.29.0-3.el9 ################################# [ 50%] Cleaning up / removing... 2:buildah-1:1.29.0-2.el9 ################################# [100%] [root@ae3bc3f4dc57 /]# buildah from ubi9 ubi9-working-container [root@ae3bc3f4dc57 /]# buildah ps CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME 37e25412c686 * 10acc174412e registry.access.redhat.com/ub... ubi9-working-container [root@ae3bc3f4dc57 /]# buildah run --isolation=chroot ubi9-working-container ls / afs bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var @dwalsh so is this only a change documentation BZ at this point? If so, where do we document this. @gnecasov heads up. We should list the current list of default capabilities. I think Red Hat could add a new article explaining the case where a container need sys_chroot. I'm assigning this to @gnecasov for the documentation additions. Gabi, please contact Giuseppe and Dan if you have questions. Gabi and Giuseppe, thanks a bunch for working through this. I'm going to close this current release as the documentation has been published with the expected behavior outlined.. Just a record. [root@kvm-01-guest10 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 9.2 (Plow) [root@kvm-01-guest10 ~]# rpm -q buildah podman containers-common kernel buildah-1.29.1-2.el9_2.x86_64 podman-4.4.1-7.el9_2.x86_64 containers-common-1-52.el9_2.x86_64 kernel-5.14.0-284.10.1.el9_2.x86_64 [root@kvm-01-guest10 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-3 Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-3... Getting image source signatures Copying blob 8fcb66e1c5fd done Copying blob f375df5edc3f done Copying config 05a32ed22e done Writing manifest to image destination Storing signatures [root@27d372de3718 /]# rpm -q buildah fuse-overlayfs containers-common buildah-1.29.0-2.el9.x86_64 fuse-overlayfs-1.10-2.el9.x86_64 containers-common-1-49.el9_1.x86_64 [root@27d372de3718 /]# buildah from ubi9 Resolved "ubi9" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf) Trying to pull registry.access.redhat.com/ubi9:latest... Getting image source signatures Checking if image destination supports signatures Copying blob 2a625e4afab5 done Copying config 9877f06ecc done Writing manifest to image destination Storing signatures ubi9-working-container [root@27d372de3718 /]# [root@27d372de3718 /]# buildah ps CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME 728e8a3e8f23 * 9877f06ecc6f registry.access.redhat.com/ub... ubi9-working-container [root@27d372de3718 /]# buildah run --isolation=chroot ubi9-working-container ls / afs bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var |