Bug 1918554
| Summary: | error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah396339746/mnt/rootfs/dev: operation not permitted | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Alex Jia <ajia> |
| Component: | fuse-overlayfs | Assignee: | Jindrich Novy <jnovy> |
| Status: | CLOSED ERRATA | QA Contact: | Alex Jia <ajia> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 8.4 | CC: | dwalsh, gscrivan, jnovy, lsm5, ypu |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 8.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | fuse-overlayfs-1.4.0-2.el8 or newer | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-05-18 15:34:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Did you run podman as root? We don't support this in rootless mode yet. (In reply to Daniel Walsh from comment #2) > Did you run podman as root? We don't support this in rootless mode yet. Yes, I ran podman as root, and I got the same issue on registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.3-20 # rpm -q podman runc crun podman-3.0.0-0.21.module+el8.4.0+9425+98db097b.x86_64 runc-1.0.0-69.rc92.module+el8.4.0+9425+98db097b.x86_64 crun-0.16-2.module+el8.4.0+9425+98db097b.x86_64 # podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.3-20 /bin/bash [root@811d639ed64b /]# rpm -q buildah fuse-overlayfs buildah-1.16.7-3.module+el8.3.1+9380+85743958.x86_64 fuse-overlayfs-1.3.0-1.module+el8.3.1+9380+85743958.x86_64 [root@811d639ed64b /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs from registry.access.redhat.com/ubi8 Getting image source signatures Copying blob d9e72d058dc5 done Copying blob cca21acb641a done Copying config 3269c37eae done Writing manifest to image destination Storing signatures ubi8-working-container [root@811d639ed64b /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run --isolation=chroot ubi8-working-container ls / error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah779829866/mnt/rootfs/dev: operation not permitted exit status 1 ERRO exit status 1 Debugging message: DEBU Running &exec.Cmd{Path:"/proc/self/exe", Args:[]string{"buildah-chroot-runtime"}, Env:[]string{"LOGLEVEL=5", "LANG=C.utf8", "HOSTNAME=811d639ed64b", "container=oci", "PWD=/", "HOME=/root", "BUILDAH_ISOLATION=chroot", "TERM=xterm", "_BUILDAH_STARTED_IN_USERNS=", "SHLVL=1", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "_=/usr/bin/buildah", "TMPDIR=/var/tmp", "_CONTAINERS_USERNS_CONFIGURED=1", "XDG_RUNTIME_DIR=/var/tmp/containers-user-0/containers"}, Dir:"/", Stdin:(*os.File)(0xc000128000), Stdout:(*os.File)(0xc000128008), Stderr:(*os.File)(0xc000128010), ExtraFiles:[]*os.File(nil), SysProcAttr:(*syscall.SysProcAttr)(nil), Process:(*os.Process)(nil), ProcessState:(*os.ProcessState)(nil), ctx:context.Context(nil), lookPathErr:error(nil), finished:false, childFiles:[]*os.File(nil), closeAfterStart:[]io.Closer(nil), closeAfterWait:[]io.Closer(nil), goroutine:[]func() error(nil), errch:(chan error)(nil), waitDone:(chan struct {})(nil)} in &unshare.Cmd{Cmd:(*exec.Cmd)(0xc000154000), UnshareFlags:0, UseNewuidmap:false, UidMappings:[]specs.LinuxIDMapping(nil), UseNewgidmap:false, GidMappings:[]specs.LinuxIDMapping(nil), GidMappingsEnableSetgroups:false, Setsid:false, Setpgrp:false, Ctty:(*os.File)(nil), OOMScoreAdj:(*int)(nil), Hook:(func(int) error)(nil)} DEBU bind mounted "/var/lib/containers/storage/overlay/dbeb78b7fc7a49a36812ddf6e4ea402e5c3062cace41427e9e7eb234506ac8a6/merged" to "/var/tmp/buildah493197107/mnt/rootfs" DEBU bind mounted "/var/lib/containers/storage/overlay-containers/9d837bfcb68e3e14af9649fb8544ad4f06cf91d0c84ecd2d2c0023c33c204a09/userdata/run/secrets" to "/var/tmp/buildah493197107/mnt/buildah-bind-target-0" error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah493197107/mnt/rootfs/dev: operation not permitted DEBU error running [ls /] in container "ubi8-working-container": exit status 1 exit status 1 DEBU shutting down the store ERRO exit status 1 This seems to be related to fuse-overlayfs using openat2 syscall which is unsupported in RHEL8 kernels (yet). Alex, before I update 8.4.0 builds, can you please retry with installed fuse-overlayfs from here? https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=34607443 Alex, if confirmed, please set qa ack+. Jindrich nice find. Alex this needs to work without specifying --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs That should be the default. If you do buildah info you should see the fuse-overlayfs in the storage options. (In reply to Daniel Walsh from comment #6) > Jindrich nice find. > > Alex this needs to work without specifying > --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs > > That should be the default. Yes, good to know this, thanks! > > If you do buildah info you should see the fuse-overlayfs in the storage > options. It works well for me when I upgraded fuse-overlayfs to 1.4.0-2 inside the buildah container. [root@kvm-06-guest09 ~]# rpm -q podman podman-3.0.0-0.21.module+el8.4.0+9425+98db097b.x86_64 [root@kvm-06-guest09 ~]# buildah info|grep -iA4 graphoption "GraphOptions": [ "overlay.mountopt=nodev,metacopy=on" ], "GraphRoot": "/var/lib/containers/storage", "GraphStatus": { [root@kvm-06-guest09 ~]# podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1... Getting image source signatures Copying blob 970900082cb8 done Copying blob 6ab905784268 done Copying blob 13990a281d31 done Copying config f7d993552e done Writing manifest to image destination Storing signatures [root@ec3971d00956 /]# rpm -q buildah fuse-overlayfs buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64 fuse-overlayfs-1.3.0-1.module+el8.3.1+9107+df0d2892.x86_64 [root@ec3971d00956 /]# buildah info|grep -iA4 graphoption "GraphOptions": [ "overlay.imagestore=/var/lib/shared", "overlay.mount_program=/usr/bin/fuse-overlayfs", "overlay.mountopt=nodev,metacopy=on" ], [root@ec3971d00956 /]# buildah from registry.access.redhat.com/ubi8 Getting image source signatures Copying blob d9e72d058dc5 done Copying blob cca21acb641a done Copying config 3269c37eae done Writing manifest to image destination Storing signatures ubi8-working-container [root@ec3971d00956 /]# buildah run --isolation=chroot ubi8-working-container ls / error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah672131359/mnt/rootfs/dev: operation not permitted exit status 1 ERRO exit status 1 [root@ec3971d00956 /]# curl -LO http://XXX/fuse-overlayfs-1.4.0-2.el8.x86_64.rpm % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 70860 100 70860 0 0 5323k 0 --:--:-- --:--:-- --:--:-- 5766k [root@ec3971d00956 /]# rpm -Uvh fuse-overlayfs-1.4.0-2.el8.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:fuse-overlayfs-1.4.0-2.el8 ################################# [ 50%] Cleaning up / removing... 2:fuse-overlayfs-1.3.0-1.module+el8################################# [100%] [root@ec3971d00956 /]# rpm -q buildah fuse-overlayfs buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64 fuse-overlayfs-1.4.0-2.el8.x86_64 [root@ec3971d00956 /]# buildah run --isolation=chroot ubi8-working-container ls / bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var Move this bug to VERIFIED status per Comment 7. Also verified this bug when only upgraded libseccomp to 2.5.1 on the host. [root@ibm-x3650m4-01-vm-16 ~]# rpm -Uvh libseccomp-2.5.1-1.el8.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:libseccomp-2.5.1-1.el8 ################################# [ 50%] Cleaning up / removing... 2:libseccomp-2.4.3-1.el8 ################################# [100%] [root@ibm-x3650m4-01-vm-16 ~]# rpm -q fuse-overlayfs buildah podman libseccomp fuse-overlayfs-1.4.0-2.module+el8.4.0+9998+ef3e9baf.x86_64 buildah-1.19.5-4.module+el8.4.0+9980+44630550.x86_64 podman-3.0.0-4.module+el8.4.0+9998+ef3e9baf.x86_64 libseccomp-2.5.1-1.el8.x86_64 [root@ibm-x3650m4-01-vm-16 ~]# podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash [root@ea001e42371a /]# rpm -q fuse-overlayfs buildah libseccomp fuse-overlayfs-1.3.0-1.module+el8.3.1+9107+df0d2892.x86_64 buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64 libseccomp-2.4.3-1.el8.x86_64 [root@ea001e42371a /]# buildah from registry.access.redhat.com/ubi8 Getting image source signatures Copying blob 6b536614e8f8 done Copying blob fdb393d8227c done Copying config 4199acc83c done Writing manifest to image destination Storing signatures ubi8-working-container [root@ea001e42371a /]# buildah run --isolation=chroot ubi8-working-container ls / bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var [root@ea001e42371a /]# exit exit Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1796 |
Description of problem: Failed to run ubi container inside buildah-container and got error like this 'error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah396339746/mnt/rootfs/dev: operation not permitted' Version-Release number of selected component (if applicable): [root@kvm-08-guest17 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 Beta (Ootpa) [root@kvm-08-guest17 ~]# rpm -q podman runc crun kernel podman-3.0.0-0.21.module+el8.4.0+9425+98db097b.x86_64 runc-1.0.0-69.rc92.module+el8.4.0+9425+98db097b.x86_64 crun-0.16-2.module+el8.4.0+9425+98db097b.x86_64 kernel-4.18.0-272.el8.dt5.x86_64 How reproducible: always Steps to Reproduce: 1. podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash 2. buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs from registry.access.redhat.com/ubi8 3. buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run --isolation=chroot ubi8-working-container ls / Actual results: [root@kvm-08-guest17 ~]# lsmod|grep fuse fuse 151552 1 [root@kvm-08-guest17 ~]# podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1... Getting image source signatures Copying blob 970900082cb8 done Copying blob 13990a281d31 done Copying blob 6ab905784268 done Copying config f7d993552e done Writing manifest to image destination Storing signatures [root@49004ecf5a2a /]# rpm -q buildah fuse-overlayfs buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64 fuse-overlayfs-1.3.0-1.module+el8.3.1+9107+df0d2892.x86_64 [root@49004ecf5a2a /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs from registry.access.redhat.com/ubi8 Getting image source signatures Copying blob d9e72d058dc5 done Copying blob cca21acb641a done Copying config 3269c37eae done Writing manifest to image destination Storing signatures ubi8-working-container [root@49004ecf5a2a /]# buildah ps CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME 3a95002afc59 * 3269c37eae33 registry.access.redhat.com/ub... ubi8-working-container [root@49004ecf5a2a /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run --isolation=chroot ubi8-working-container ls / error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah396339746/mnt/rootfs/dev: operation not permitted exit status 1 ERRO exit status 1 Expected results: fix it. Additional info: