Bug 2066138

Summary: failed to open /dev/fuse: Operation not permitted
Product: Red Hat Enterprise Linux 8 Reporter: Alex Jia <ajia>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Alex Jia <ajia>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.7CC: bbaude, ddarrah, dwalsh, gscrivan, jligon, jnovy, lsm5, mbearer, mheon, pthomas, tsweeney, umohnani, ypu
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2057243 Environment:
Last Closed: 2023-11-14 15:27:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 3 Alex Jia 2022-03-22 12:16:45 UTC
This bug still exists on podman-4.0.2-2.module+el8.7.0+14506+57abd10a.

To provision a fresh 8.7 w/ RHEL-8.7.0-20220321.0 distro in the beaker.

[root@sweetpig-18 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 Beta (Ootpa)

To install newest container-tools-rhel8-8070020220318113725-39077419 module then test.

[root@sweetpig-18 ~]# rpm -q podman fuse-overlayfs fuse3 runc systemd kernel
podman-4.0.2-2.module+el8.7.0+14506+57abd10a.x86_64                       
fuse-overlayfs-1.8.2-1.module+el8.7.0+14506+57abd10a.x86_64
fuse3-3.3.0-15.el8.x86_64
runc-1.0.3-3.module+el8.7.0+14506+57abd10a.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-372.2.1.el8.x86_64

[root@sweetpig-18 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-5
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-5...
Getting image source signatures
Copying blob 2907092333e1 done  
Copying blob a6fd35289163 done  
Copying blob 76ea87ceff0d done  
Copying config eda4e8afab done  
Writing manifest to image destination
Storing signatures
[root@e54416e78680 /]# rpm -q buildah fuse-overlayfs
buildah-1.24.2-2.module+el8.6.0+14488+6524fb7f.x86_64
fuse-overlayfs-1.8.2-1.module+el8.6.0+14488+6524fb7f.x86_64
[root@e54416e78680 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob c530010fb61c done  
Copying blob 3de00bb8554b done  
Copying config 552ac8ae42 done  
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@e54416e78680 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
ba8d84ddbc0c     *     552ac8ae4291 registry.access.redhat.com/ub... ubi8-working-container
[root@e54416e78680 /]# buildah run  --isolation=chroot ubi8-working-container ls
ERRO[0000] Unmounting /var/lib/containers/storage/overlay/720f53af8b36ccdf25187e696b086a8a3696a9688ddd7884dcc392cc9598b54d/merged: invalid argument 
error mounting container "ba8d84ddbc0cc7aaafba41ac06e5ec9acd676fab3d0ee03ce2a7228bcc37ae0c": error mounting build container "ba8d84ddbc0cc7aaafba41ac06e5ec9acd676fab3d0ee03ce2a7228bcc37ae0c": creating overlay mount to /var/lib/containers/storage/overlay/720f53af8b36ccdf25187e696b086a8a3696a9688ddd7884dcc392cc9598b54d/merged, mount_data="nodev,lowerdir=/var/lib/containers/storage/overlay/l/Z4KCVKAEG7VLE7NE3E5TIZQCHH:/var/lib/containers/storage/overlay/l/PB6XF26GR5JIH76XEBF2EOWSD5,upperdir=/var/lib/containers/storage/overlay/720f53af8b36ccdf25187e696b086a8a3696a9688ddd7884dcc392cc9598b54d/diff,workdir=/var/lib/containers/storage/overlay/720f53af8b36ccdf25187e696b086a8a3696a9688ddd7884dcc392cc9598b54d/work,volatile": using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 125

Comment 4 Daniel Walsh 2022-03-22 14:44:50 UTC
This is not working on Fedora either.  I am not sure what is going wrong.

Giuseppe any ideas?

Comment 5 Giuseppe Scrivano 2022-03-23 16:31:14 UTC
Fixed upstream with d2f77c256836e8a1a562d56eed7d92fdb18a2f76

Comment 6 Alex Jia 2022-03-24 11:05:31 UTC
(In reply to Giuseppe Scrivano from comment #5)
> Fixed upstream with d2f77c256836e8a1a562d56eed7d92fdb18a2f76

This patch works well for me on RHEL-8.6.

[root@kvm-06-guest34 podman]# git show d2f77c25 --no-patch
commit d2f77c256836e8a1a562d56eed7d92fdb18a2f76
Author: Hironori Shiina <shiina.hironori.com>
Date:   Thu Mar 3 21:39:14 2022 -0500

    Set default rule at the head of device configuration
    
    The default rule should be set at the head of device configuration.
    Otherwise, rules for user devices are overridden by the default rule so
    that any access to the user devices are denied.
    
    Signed-off-by: Hironori Shiina <shiina.hironori.com>


[root@kvm-06-guest34 podman]# ./bin/podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-5
[root@7779acb7d81c /]# rpm -q buildah fuse-overlayfs
buildah-1.24.2-2.module+el8.6.0+14488+6524fb7f.x86_64
fuse-overlayfs-1.8.2-1.module+el8.6.0+14488+6524fb7f.x86_64
[root@7779acb7d81c /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 3de00bb8554b done  
Copying blob c530010fb61c done  
Copying config 552ac8ae42 done  
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@7779acb7d81c /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
e05ba45ab055     *     552ac8ae4291 registry.access.redhat.com/ub... ubi8-working-container
[root@7779acb7d81c /]# buildah run  --isolation=chroot ubi8-working-container ls
bin  boot  dev	etc  home  lib	lib64  lost+found  media  mnt  opt  proc  root	run  sbin  srv	sys  tmp  usr  var

Comment 7 Alex Jia 2022-03-24 11:07:43 UTC
(In reply to Alex Jia from comment #6)
> (In reply to Giuseppe Scrivano from comment #5)
> > Fixed upstream with d2f77c256836e8a1a562d56eed7d92fdb18a2f76
> 
> This patch works well for me on RHEL-8.6.
> 
s/RHEL-8.6/RHEL-8.7/

Comment 8 Tom Sweeney 2022-03-24 13:18:19 UTC
Assigning to @jnovy for any further BZ or packaging needs.

Comment 9 Jindrich Novy 2022-03-28 12:19:29 UTC
This fix will be present in 8.7.0 in the next release of podman.

Comment 10 Alex Jia 2022-04-02 07:44:37 UTC
This bug has been verified on podman-4.0.3-1.module+el8.7.0+14675+ab3d5383.

[root@hpe-dl380pgen8-02-vm-14 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 Beta (Ootpa)

[root@hpe-dl380pgen8-02-vm-14 ~]# rpm -q podman runc systemd kernel
podman-4.0.3-1.module+el8.7.0+14675+ab3d5383.x86_64
runc-1.0.3-3.module+el8.7.0+14675+ab3d5383.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-375.el8.x86_64

[root@hpe-dl380pgen8-02-vm-14 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-6
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-6...
Getting image source signatures
Copying blob 9a58214aa887 done  
Copying blob 01ce70940a94 done  
Copying blob 1aed92981855 done  
Copying config e13113c970 done  
Writing manifest to image destination
Storing signatures
[root@8b0937255fed /]# rpm -q buildah fuse-overlayfs
buildah-1.24.2-2.module+el8.6.0+14488+6524fb7f.x86_64
fuse-overlayfs-1.8.2-1.module+el8.6.0+14488+6524fb7f.x86_64
[root@8b0937255fed /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4eef1fa1f1c1 done  
Copying blob eb24191cef20 done  
Copying config c54243b588 done  
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@8b0937255fed /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
dcf34bfd5fdf     *     c54243b58814 registry.access.redhat.com/ub... ubi8-working-container
[root@8b0937255fed /]# buildah run  --isolation=chroot ubi8-working-container ls
bin  boot  dev	etc  home  lib	lib64  lost+found  media  mnt  opt  proc  root	run  sbin  srv	sys  tmp  usr  var
[root@8b0937255fed /]# exit
exit
[root@hpe-dl380pgen8-02-vm-14 ~]# echo $?
0

Comment 11 Tom Sweeney 2022-08-22 21:03:10 UTC
@jnovy can this be closed?

Comment 12 Jindrich Novy 2022-08-23 07:24:52 UTC
Not yet, this waits to be released with 8.7.0.

Comment 17 errata-xmlrpc 2023-11-14 15:27:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6939