RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1918554 - error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah396339746/mnt/rootfs/dev: operation not permitted
Summary: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/build...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: fuse-overlayfs
Version: 8.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-21 02:55 UTC by Alex Jia
Modified: 2021-05-18 15:35 UTC (History)
5 users (show)

Fixed In Version: fuse-overlayfs-1.4.0-2.el8 or newer
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:34:30 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Alex Jia 2021-01-21 02:55:15 UTC 
Description of problem:
Failed to run ubi container inside buildah-container and got error like this 'error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah396339746/mnt/rootfs/dev: operation not permitted'

Version-Release number of selected component (if applicable):
[root@kvm-08-guest17 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)

[root@kvm-08-guest17 ~]# rpm -q podman runc crun kernel
podman-3.0.0-0.21.module+el8.4.0+9425+98db097b.x86_64
runc-1.0.0-69.rc92.module+el8.4.0+9425+98db097b.x86_64
crun-0.16-2.module+el8.4.0+9425+98db097b.x86_64
kernel-4.18.0-272.el8.dt5.x86_64

How reproducible:
always

Steps to Reproduce:
1. podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash
2. buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs from registry.access.redhat.com/ubi8
3. buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run --isolation=chroot ubi8-working-container ls /

Actual results:

[root@kvm-08-guest17 ~]# lsmod|grep fuse
fuse                  151552  1
[root@kvm-08-guest17 ~]# podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1...
Getting image source signatures
Copying blob 970900082cb8 done
Copying blob 13990a281d31 done
Copying blob 6ab905784268 done
Copying config f7d993552e done
Writing manifest to image destination
Storing signatures
[root@49004ecf5a2a /]# rpm -q buildah fuse-overlayfs
buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64
fuse-overlayfs-1.3.0-1.module+el8.3.1+9107+df0d2892.x86_64
[root@49004ecf5a2a /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs from registry.access.redhat.com/ubi8
Getting image source signatures
Copying blob d9e72d058dc5 done
Copying blob cca21acb641a done
Copying config 3269c37eae done
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@49004ecf5a2a /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
3a95002afc59     *     3269c37eae33 registry.access.redhat.com/ub... ubi8-working-container
[root@49004ecf5a2a /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run --isolation=chroot ubi8-working-container ls /
error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah396339746/mnt/rootfs/dev: operation not permitted
                                                                                                                                                          exit status 1
ERRO exit status 1

Expected results:
fix it.

Additional info:
Comment 2 Daniel Walsh 2021-01-28 21:51:54 UTC 
Did you run podman as root?  We don't support this in rootless mode yet.
Comment 3 Alex Jia 2021-01-29 10:39:46 UTC 
(In reply to Daniel Walsh from comment #2)
> Did you run podman as root?  We don't support this in rootless mode yet.

Yes, I ran podman as root, and I got the same issue on registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.3-20

# rpm -q podman runc crun
podman-3.0.0-0.21.module+el8.4.0+9425+98db097b.x86_64
runc-1.0.0-69.rc92.module+el8.4.0+9425+98db097b.x86_64
crun-0.16-2.module+el8.4.0+9425+98db097b.x86_64

# podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.3-20 /bin/bash
[root@811d639ed64b /]# rpm -q buildah fuse-overlayfs
buildah-1.16.7-3.module+el8.3.1+9380+85743958.x86_64
fuse-overlayfs-1.3.0-1.module+el8.3.1+9380+85743958.x86_64
[root@811d639ed64b /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs from registry.access.redhat.com/ubi8
Getting image source signatures
Copying blob d9e72d058dc5 done
Copying blob cca21acb641a done
Copying config 3269c37eae done
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@811d639ed64b /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run --isolation=chroot ubi8-working-container ls /
error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah779829866/mnt/rootfs/dev: operation not permitted
                                                                                                                                                          exit status 1
ERRO exit status 1

Debugging message:

DEBU Running &exec.Cmd{Path:"/proc/self/exe", Args:[]string{"buildah-chroot-runtime"}, Env:[]string{"LOGLEVEL=5", "LANG=C.utf8", "HOSTNAME=811d639ed64b", "container=oci", "PWD=/", "HOME=/root", "BUILDAH_ISOLATION=chroot", "TERM=xterm", "_BUILDAH_STARTED_IN_USERNS=", "SHLVL=1", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "_=/usr/bin/buildah", "TMPDIR=/var/tmp", "_CONTAINERS_USERNS_CONFIGURED=1", "XDG_RUNTIME_DIR=/var/tmp/containers-user-0/containers"}, Dir:"/", Stdin:(*os.File)(0xc000128000), Stdout:(*os.File)(0xc000128008), Stderr:(*os.File)(0xc000128010), ExtraFiles:[]*os.File(nil), SysProcAttr:(*syscall.SysProcAttr)(nil), Process:(*os.Process)(nil), ProcessState:(*os.ProcessState)(nil), ctx:context.Context(nil), lookPathErr:error(nil), finished:false, childFiles:[]*os.File(nil), closeAfterStart:[]io.Closer(nil), closeAfterWait:[]io.Closer(nil), goroutine:[]func() error(nil), errch:(chan error)(nil), waitDone:(chan struct {})(nil)} in &unshare.Cmd{Cmd:(*exec.Cmd)(0xc000154000), UnshareFlags:0, UseNewuidmap:false, UidMappings:[]specs.LinuxIDMapping(nil), UseNewgidmap:false, GidMappings:[]specs.LinuxIDMapping(nil), GidMappingsEnableSetgroups:false, Setsid:false, Setpgrp:false, Ctty:(*os.File)(nil), OOMScoreAdj:(*int)(nil), Hook:(func(int) error)(nil)}
                                                                              DEBU bind mounted "/var/lib/containers/storage/overlay/dbeb78b7fc7a49a36812ddf6e4ea402e5c3062cace41427e9e7eb234506ac8a6/merged" to "/var/tmp/buildah493197107/mnt/rootfs"
                                                                          DEBU bind mounted "/var/lib/containers/storage/overlay-containers/9d837bfcb68e3e14af9649fb8544ad4f06cf91d0c84ecd2d2c0023c33c204a09/userdata/run/secrets" to "/var/tmp/buildah493197107/mnt/buildah-bind-target-0"
                                                                                                              error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah493197107/mnt/rootfs/dev: operation not permitted
                                                                                          DEBU error running [ls /] in container "ubi8-working-container": exit status 1
exit status 1
DEBU shutting down the store
ERRO exit status 1
Comment 4 Jindrich Novy 2021-01-29 11:07:26 UTC 
This seems to be related to fuse-overlayfs using openat2 syscall which is unsupported in RHEL8 kernels (yet).

Alex, before I update 8.4.0 builds, can you please retry with installed fuse-overlayfs from here? https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=34607443
Comment 5 Jindrich Novy 2021-01-29 11:08:27 UTC 
Alex, if confirmed, please set qa ack+.
Comment 6 Daniel Walsh 2021-01-30 12:14:41 UTC 
Jindrich nice find.

Alex this needs to work without specifying --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs

That should be the default.

If you do buildah info you should see the fuse-overlayfs in the storage options.
Comment 7 Alex Jia 2021-02-01 04:00:33 UTC 
(In reply to Daniel Walsh from comment #6)
> Jindrich nice find.
> 
> Alex this needs to work without specifying
> --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs
> 
> That should be the default.

Yes, good to know this, thanks!

> 
> If you do buildah info you should see the fuse-overlayfs in the storage
> options.

It works well for me when I upgraded fuse-overlayfs to 1.4.0-2 inside the buildah container.

[root@kvm-06-guest09 ~]# rpm -q podman
podman-3.0.0-0.21.module+el8.4.0+9425+98db097b.x86_64
[root@kvm-06-guest09 ~]# buildah info|grep -iA4 graphoption
        "GraphOptions": [
            "overlay.mountopt=nodev,metacopy=on"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {

[root@kvm-06-guest09 ~]# podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1...
Getting image source signatures
Copying blob 970900082cb8 done
Copying blob 6ab905784268 done
Copying blob 13990a281d31 done
Copying config f7d993552e done
Writing manifest to image destination
Storing signatures

[root@ec3971d00956 /]# rpm -q buildah fuse-overlayfs
buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64
fuse-overlayfs-1.3.0-1.module+el8.3.1+9107+df0d2892.x86_64

[root@ec3971d00956 /]# buildah info|grep -iA4 graphoption
        "GraphOptions": [
            "overlay.imagestore=/var/lib/shared",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,metacopy=on"
        ],

[root@ec3971d00956 /]# buildah from registry.access.redhat.com/ubi8
Getting image source signatures
Copying blob d9e72d058dc5 done
Copying blob cca21acb641a done
Copying config 3269c37eae done
Writing manifest to image destination
Storing signatures
ubi8-working-container

[root@ec3971d00956 /]# buildah run --isolation=chroot ubi8-working-container ls /
error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah672131359/mnt/rootfs/dev: operation not permitted
                                                                                                                                                          exit status 1
ERRO exit status 1

[root@ec3971d00956 /]# curl -LO http://XXX/fuse-overlayfs-1.4.0-2.el8.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70860  100 70860    0     0  5323k      0 --:--:-- --:--:-- --:--:-- 5766k
[root@ec3971d00956 /]# rpm -Uvh fuse-overlayfs-1.4.0-2.el8.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:fuse-overlayfs-1.4.0-2.el8       ################################# [ 50%]
Cleaning up / removing...
   2:fuse-overlayfs-1.3.0-1.module+el8################################# [100%]

[root@ec3971d00956 /]# rpm -q buildah fuse-overlayfs
buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64
fuse-overlayfs-1.4.0-2.el8.x86_64

[root@ec3971d00956 /]# buildah run --isolation=chroot ubi8-working-container ls /
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
Comment 12 Alex Jia 2021-02-01 17:13:22 UTC 
Move this bug to VERIFIED status per Comment 7.
Comment 13 Alex Jia 2021-02-19 11:32:16 UTC 
Also verified this bug when only upgraded libseccomp to 2.5.1 on the host.

[root@ibm-x3650m4-01-vm-16 ~]# rpm -Uvh libseccomp-2.5.1-1.el8.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:libseccomp-2.5.1-1.el8           ################################# [ 50%]
Cleaning up / removing...
   2:libseccomp-2.4.3-1.el8           ################################# [100%]

[root@ibm-x3650m4-01-vm-16 ~]# rpm -q fuse-overlayfs buildah podman libseccomp
fuse-overlayfs-1.4.0-2.module+el8.4.0+9998+ef3e9baf.x86_64
buildah-1.19.5-4.module+el8.4.0+9980+44630550.x86_64
podman-3.0.0-4.module+el8.4.0+9998+ef3e9baf.x86_64
libseccomp-2.5.1-1.el8.x86_64

[root@ibm-x3650m4-01-vm-16 ~]# podman run --name rhel8-buildah --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.4-1 /bin/bash
[root@ea001e42371a /]# rpm -q fuse-overlayfs buildah libseccomp
fuse-overlayfs-1.3.0-1.module+el8.3.1+9107+df0d2892.x86_64
buildah-1.16.7-1.module+el8.3.1+9107+df0d2892.x86_64
libseccomp-2.4.3-1.el8.x86_64

[root@ea001e42371a /]# buildah from registry.access.redhat.com/ubi8
Getting image source signatures
Copying blob 6b536614e8f8 done
Copying blob fdb393d8227c done
Copying config 4199acc83c done
Writing manifest to image destination
Storing signatures
ubi8-working-container

[root@ea001e42371a /]# buildah run --isolation=chroot ubi8-working-container ls /
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@ea001e42371a /]# exit
exit
Comment 15 errata-xmlrpc 2021-05-18 15:34:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1796
Note You need to log in before you can comment on or make changes to this bug.