Bug 1867447

Description of problem:
Failed to run ubi8-working-container inside rhel8-buildah:8.3-5, and got error like this "error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah050538105/mnt/rootfs/dev: operation not permitted
                                                                                                                                                          exit status 1
ERRO exit status 1"

For details, please see 'Additional info' section.

Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1. modprobe fuse 
2. podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.3-5 /bin/bash
3. buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs from registry.access.redhat.com/ubi8
4. buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run  --isolation=chroot ubi8-working-container ls /

Actual results:
[root@2e79b51df22c /]# buildah --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run  --isolation=chroot ubi8-working-container ls /
error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah050538105/mnt/rootfs/dev: operation not permitted
                                                                                                                                                          exit status 1
ERRO exit status 1

Expected results:
can successfully run buildah-container 8.3

Additional info:
[root@2e79b51df22c /]# buildah --debug --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run  --isolation=chroot ubi8-working-container ls /
DEBU running [buildah-in-a-user-namespace --debug --storage-opt=overlay.mount_program=/usr/bin/fuse-overlayfs run --isolation=chroot ubi8-working-container ls /] with environment [LANG=C.utf8 HOSTNAME=2e79b51df22c container=podman PWD=/ HOME=/root BUILDAH_ISOLATION=chroot TERM=xterm _BUILDAH_STARTED_IN_USERNS= SHLVL=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin _=/usr/bin/buildah TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1], UID map [{ContainerID:0 HostID:0 Size:4294967295}], and GID map [{ContainerID:0 HostID:0 Size:4294967295}]
DEBU [graphdriver] trying provided driver "overlay"
DEBU overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU using "/var/tmp/buildah117605080" to hold bundle data
DEBU Resources: &buildah.CommonBuildOptions{AddHost:[]string{}, CgroupParent:"", CPUPeriod:0x0, CPUQuota:0, CPUShares:0x0, CPUSetCPUs:"", CPUSetMems:"", HTTPProxy:true, Memory:0, DNSSearch:[]string{}, DNSServers:[]string{}, DNSOptions:[]string{}, MemorySwap:0, LabelOpts:[]string(nil), SeccompProfilePath:"/usr/share/containers/seccomp.json", ApparmorProfile:"", ShmSize:"65536k", Ulimit:[]string{"nproc=4194304:4194304"}, Volumes:[]string{}}
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/PW7QKHKPMCJNI5EIX6AJ5ZCCKQ:/var/lib/containers/storage/overlay/l/IK2NJXKEPOCZRYOSSD745DKWMH,upperdir=/var/lib/containers/storage/overlay/9f883df0203ec9fa2823c4a62eab7529a300347545bc421b219acc44d231dc5a/diff,workdir=/var/lib/containers/storage/overlay/9f883df0203ec9fa2823c4a62eab7529a300347545bc421b219acc44d231dc5a/work
DEBU stdio is a terminal, defaulting to using a terminal
DEBU ensuring working directory "/var/lib/containers/storage/overlay/9f883df0203ec9fa2823c4a62eab7529a300347545bc421b219acc44d231dc5a/merged" exists
DEBU /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU config = {"ociVersion":"1.0.2-dev","process":{"terminal":true,"user":{"uid":0,"gid":0},"args":["ls","/"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=oci","HOSTNAME=eba898107bd8","HOME=/"],"cwd":"/","capabilities":{"bounding":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"effective":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"inheritable":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"permitted":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"ambient":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"]},"rlimits":[{"type":"RLIMIT_NOFILE","hard":1024,"soft":1024},{"type":"RLIMIT_NPROC","hard":4194304,"soft":4194304}]},"root":{"path":"/var/lib/containers/storage/overlay/9f883df0203ec9fa2823c4a62eab7529a300347545bc421b219acc44d231dc5a/merged"},"hostname":"eba898107bd8","mounts":[{"destination":"/run/secrets","type":"bind","source":"/var/lib/containers/storage/overlay-containers/74e6c88ee22844d16a9ec39f3e9f7a8a60a1c7a70b6e1e54ab1e91152c27376b/userdata/run/secrets","options":["bind","rprivate"]},{"destination":"/etc/resolv.conf","type":"bind","source":"/var/tmp/buildah117605080/resolv.conf","options":["rbind","nobuildahbind"]},{"destination":"/run/.containerenv","type":"bind","source":"/var/tmp/buildah117605080/run/.containerenv","options":["rbind","nobuildahbind"]},{"destination":"/etc/hosts","type":"bind","source":"/var/tmp/buildah117605080/hosts","options":["rbind","nobuildahbind"]},{"destination":"/proc","type":"proc","source":"proc","options":["nosuid","noexec","nodev"]},{"destination":"/dev","type":"tmpfs","source":"tmpfs","options":["nosuid","strictatime","mode=755","size=65536k"]},{"destination":"/dev/pts","type":"devpts","source":"devpts","options":["nosuid","noexec","newinstance","ptmxmode=0666","mode=0620","gid=5"]},{"destination":"/dev/shm","type":"tmpfs","source":"shm","options":["nosuid","noexec","nodev","mode=1777","size=65536k"]},{"destination":"/dev/mqueue","type":"mqueue","source":"mqueue","options":["nosuid","noexec","nodev"]},{"destination":"/sys","type":"bind","source":"/sys","options":["nobuildahbind","rbind","nosuid","noexec","nodev","ro"]}],"linux":{"resources":{"devices":[{"allow":false,"access":"rwm"}]},"namespaces":[{"type":"pid"},{"type":"ipc"},{"type":"uts"},{"type":"mount"}],"seccomp":{"defaultAction":"SCMP_ACT_ERRNO","architectures":["SCMP_ARCH_X86_64","SCMP_ARCH_X86","SCMP_ARCH_X32"],"syscalls":[{"names":["accept","accept4","access","adjtimex","alarm","bind","brk","capget","capset","chdir","chmod","chown","chown32","clock_adjtime","clock_getres","clock_gettime","clock_nanosleep","clone","close","connect","copy_file_range","creat","dup","dup2","dup3","epoll_create","epoll_create1","epoll_ctl","epoll_ctl_old","epoll_pwait","epoll_wait","epoll_wait_old","eventfd","eventfd2","execve","execveat","exit","exit_group","faccessat","fadvise64","fadvise64_64","fallocate","fanotify_mark","fchdir","fchmod","fchmodat","fchown","fchown32","fchownat","fcntl","fcntl64","fdatasync","fgetxattr","flistxattr","flock","fork","fremovexattr","fsetxattr","fstat","fstat64","fstatat64","fstatfs","fstatfs64","fsync","ftruncate","ftruncate64","futex","futimesat","getcpu","getcwd","getdents","getdents64","getegid","getegid32","geteuid","geteuid32","getgid","getgid32","getgroups","getgroups32","getitimer","getpeername","getpgid","getpgrp","getpid","getppid","getpriority","getrandom","getresgid","getresgid32","getresuid","getresuid32","getrlimit","get_robust_list","getrusage","getsid","getsockname","getsockopt","get_thread_area","gettid","gettimeofday","getuid","getuid32","getxattr","inotify_add_watch","inotify_init","inotify_init1","inotify_rm_watch","io_cancel","ioctl","io_destroy","io_getevents","ioprio_get","ioprio_set","io_setup","io_submit","ipc","keyctl","kill","lchown","lchown32","lgetxattr","link","linkat","listen","listxattr","llistxattr","_llseek","lremovexattr","lseek","lsetxattr","lstat","lstat64","madvise","memfd_create","mincore","mkdir","mkdirat","mknod","mknodat","mlock","mlock2","mlockall","mmap","mmap2","mprotect","mq_getsetattr","mq_notify","mq_open","mq_timedreceive","mq_timedsend","mq_unlink","mremap","msgctl","msgget","msgrcv","msgsnd","msync","munlock","munlockall","munmap","nanosleep","newfstatat","_newselect","open","openat","pause","pipe","pipe2","pivot_root","poll","ppoll","prctl","pread64","preadv","preadv2","prlimit64","pselect6","pwrite64","pwritev","pwritev2","read","readahead","readlink","readlinkat","readv","recv","recvfrom","recvmmsg","recvmsg","remap_file_pages","removexattr","rename","renameat","renameat2","restart_syscall","rmdir","rt_sigaction","rt_sigpending","rt_sigprocmask","rt_sigqueueinfo","rt_sigreturn","rt_sigsuspend","rt_sigtimedwait","rt_tgsigqueueinfo","sched_getaffinity","sched_getattr","sched_getparam","sched_get_priority_max","sched_get_priority_min","sched_getscheduler","sched_rr_get_interval","sched_setaffinity","sched_setattr","sched_setparam","sched_setscheduler","sched_yield","seccomp","select","semctl","semget","semop","semtimedop","send","sendfile","sendfile64","sendmmsg","sendmsg","sendto","setfsgid","setfsgid32","setfsuid","setfsuid32","setgid","setgid32","setgroups","setgroups32","setitimer","setpgid","setpriority","setregid","setregid32","setresgid","setresgid32","setresuid","setresuid32","setreuid","setreuid32","setrlimit","set_robust_list","setsid","setsockopt","set_thread_area","set_tid_address","setuid","setuid32","setxattr","shmat","shmctl","shmdt","shmget","shutdown","sigaltstack","signalfd","signalfd4","sigreturn","socket","socketcall","socketpair","splice","stat","stat64","statfs","statfs64","statx","symlink","symlinkat","sync","sync_file_range","syncfs","sysinfo","syslog","tee","tgkill","time","timer_create","timer_delete","timerfd_create","timerfd_gettime","timerfd_settime","timer_getoverrun","timer_gettime","timer_settime","times","tkill","truncate","truncate64","ugetrlimit","umask","uname","unlink","unlinkat","utime","utimensat","utimes","vfork","vmsplice","wait4","waitid","waitpid","write","writev","mount","umount2","reboot","name_to_handle_at","unshare"],"action":"SCMP_ACT_ALLOW"},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":0,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":8,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131072,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131080,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":4294967295,"op":"SCMP_CMP_EQ"}]},{"names":["arch_prctl"],"action":"SCMP_ACT_ALLOW"},{"names":["modify_ldt"],"action":"SCMP_ACT_ALLOW"},{"names":["clone"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":2080505856,"op":"SCMP_CMP_MASKED_EQ"}]},{"names":["chroot"],"action":"SCMP_ACT_ALLOW"}]},"maskedPaths":["/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware"],"readonlyPaths":["/proc/asound","/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]}}
DEBU Running &exec.Cmd{Path:"/proc/self/exe", Args:[]string{"buildah-chroot-runtime"}, Env:[]string{"LOGLEVEL=5", "LANG=C.utf8", "HOSTNAME=2e79b51df22c", "container=podman", "PWD=/", "HOME=/root", "BUILDAH_ISOLATION=chroot", "TERM=xterm", "_BUILDAH_STARTED_IN_USERNS=", "SHLVL=1", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "_=/usr/bin/buildah", "TMPDIR=/var/tmp", "_CONTAINERS_USERNS_CONFIGURED=1", "XDG_RUNTIME_DIR=/run/user/0/containers"}, Dir:"/", Stdin:(*os.File)(0xc000010010), Stdout:(*os.File)(0xc000010018), Stderr:(*os.File)(0xc000010020), ExtraFiles:[]*os.File(nil), SysProcAttr:(*syscall.SysProcAttr)(nil), Process:(*os.Process)(nil), ProcessState:(*os.ProcessState)(nil), ctx:context.Context(nil), lookPathErr:error(nil), finished:false, childFiles:[]*os.File(nil), closeAfterStart:[]io.Closer(nil), closeAfterWait:[]io.Closer(nil), goroutine:[]func() error(nil), errch:(chan error)(nil), waitDone:(chan struct {})(nil)} in &unshare.Cmd{Cmd:(*exec.Cmd)(0xc00015e160), UnshareFlags:0, UseNewuidmap:false, UidMappings:[]specs.LinuxIDMapping(nil), UseNewgidmap:false, GidMappings:[]specs.LinuxIDMapping(nil), GidMappingsEnableSetgroups:false, Setsid:false, Setpgrp:false, Ctty:(*os.File)(nil), OOMScoreAdj:(*int)(nil), Hook:(func(int) error)(nil)}
                                                                                                      DEBU bind mounted "/var/lib/containers/storage/overlay/9f883df0203ec9fa2823c4a62eab7529a300347545bc421b219acc44d231dc5a/merged" to "/var/tmp/buildah117605080/mnt/rootfs"
                                                                           DEBU bind mounted "/var/lib/containers/storage/overlay-containers/74e6c88ee22844d16a9ec39f3e9f7a8a60a1c7a70b6e1e54ab1e91152c27376b/userdata/run/secrets" to "/var/tmp/buildah117605080/mnt/buildah-bind-target-0"
                                                                                        error running subprocess: error bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah117605080/mnt/rootfs/dev: operation not permitted
                                             DEBU Error unmounting /var/lib/containers/storage/overlay/9f883df0203ec9fa2823c4a62eab7529a300347545bc421b219acc44d231dc5a/merged with fusermount3 - exec: "fusermount3": executable file not found in $PATH
DEBU Error unmounting /var/lib/containers/storage/overlay/9f883df0203ec9fa2823c4a62eab7529a300347545bc421b219acc44d231dc5a/merged with fusermount - exec: "fusermount": executable file not found in $PATH
DEBU error running [ls /] in container "ubi8-working-container": exit status 1
exit status 1
ERRO exit status 1

[root@2e79b51df22c /]# rpm -q buildah
buildah-1.15.0-1.module+el8.3.0+7084+c16098dd.x86_64

[root@2e79b51df22c /]# rpm -qa|grep fuse
fuse3-libs-3.2.1-12.el8.x86_64
fuse-overlayfs-1.1.1-1.module+el8.3.0+7121+472bc0cf.x86_64