Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2057337

Summary:	failed to open /dev/fuse: Operation not permitted
Product:	Red Hat Enterprise Linux 9	Reporter:	Alex Jia <ajia>
Component:	buildah-container	Assignee:	Jindrich Novy <jnovy>
Status:	CLOSED ERRATA	QA Contact:	Alex Jia <ajia>
Severity:	high	Docs Contact:	Michelle Bearer <mbearer>
Priority:	unspecified
Version:	9.0	CC:	atomic-bugs, jnovy, mbearer, tsweeney, ypu
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2057243	Environment:
Last Closed:	2022-05-18 02:50:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2057243
Bug Blocks:	2056952

Description Alex Jia 2022-02-23 08:56:01 UTC

+++ This bug was initially created as a clone of Bug #2057243 +++

Description of problem:
Failed to run embedded container inside the rhel8-buildah:8.6-2 and got error like this "xxx using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted xxx".

Version-Release number of selected component (if applicable):
[root@hpe-dl380pgen8-02-vm-9 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)
[root@hpe-dl380pgen8-02-vm-9 ~]# rpm -q podman runc systemd kernel
podman-4.0.0-0.29.module+el8.6.0+14295+adafbc4c.x86_64
runc-1.0.3-1.module+el8.6.0+14295+adafbc4c.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-367.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-2
2. buildah from registry.access.redhat.com/ubi8
3. buildah run --isolation=chroot ubi8-working-container ls 

Actual results:
[root@hpe-dl380pgen8-02-vm-9 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-2
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-2...
Getting image source signatures
Copying blob 28ced426e3f1 done
Copying blob 0b6f949925e2 done
Copying blob e79d3f31aa4c done
Copying config 3bfe906ec6 done
Writing manifest to image destination
Storing signatures
[root@019326d5da82 /]# rpm -q buildah fuse-overlayfs
buildah-1.23.1-3.module+el8.6.0+13412+2981c13c.x86_64
fuse-overlayfs-1.8-1.module+el8.6.0+13730+5dcf3a04.x86_64
[root@019326d5da82 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 5dcbdc60ea6b done
Copying blob 8671113e1c57 done
Copying config b81e86a2cb done
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@019326d5da82 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
1132ea4bdd51     *     b81e86a2cb9a registry.access.redhat.com/ub... ubi8-working-container
[root@019326d5da82 /]# buildah run --isolation=chroot ubi8-working-container ls /
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/merged: invalid argument
error mounting container "1132ea4bdd51b49feb3450f4bada455ff41cf582cfc7f4fa14e6c3fffa40280f": error mounting build container "1132ea4bdd51b49feb3450f4bada455ff41cf582cfc7f4fa14e6c3fffa40280f": error creating overlay mount to /var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/R6BKACDZ4WVQTYCL7JE3ISB3HT:/var/lib/containers/storage/overlay/l/273NNQHV56HG4M2MXQARYMRFY5,upperdir=/var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/diff,workdir=/var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/work,volatile": using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 125

Expected results:
fix it.

Additional info:

Comment 1 Alex Jia 2022-02-23 08:56:52 UTC

[root@ibm-x3250m6-04 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.0.0-3
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.0.0-3...
Getting image source signatures
Copying blob 1fb14c654ae4 done
Copying blob 52663ea9d10c done
Copying blob 3f5eb906db0f done
Copying config 3694df9b3a done
Writing manifest to image destination
Storing signatures
[root@b628756870c3 /]# rpm -q buildah fuse-overlayfs
buildah-1.24.1-1.el9.x86_64
fuse-overlayfs-1.8.2-1.el9.x86_64
[root@b628756870c3 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 8671113e1c57 done
Copying blob 5dcbdc60ea6b done
Copying config b81e86a2cb done
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@b628756870c3 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
c701e405189e     *     b81e86a2cb9a registry.access.redhat.com/ub... ubi8-working-container
[root@b628756870c3 /]# buildah run --isolation=chroot ubi8-working-container ls /
ERRO[0000] Unmounting /var/lib/containers/storage/overlay/1b9026475244313d5b30294d70b79385cc00cac26ffda7478852ca46ee71112e/merged: invalid argument
error mounting container "c701e405189ea6cd3f49a4a71f61a28892d6a0cb48da2cc633de25cb8b5c3a52": error mounting build container "c701e405189ea6cd3f49a4a71f61a28892d6a0cb48da2cc633de25cb8b5c3a52": creating overlay mount to /var/lib/containers/storage/overlay/1b9026475244313d5b30294d70b79385cc00cac26ffda7478852ca46ee71112e/merged, mount_data="nodev,lowerdir=/var/lib/containers/storage/overlay/l/BVYS7EFL3LJ5Z43OVU5UDQGBDH:/var/lib/containers/storage/overlay/l/LP6Q5UMIDRJ5GSD4XYMLZBOWGO,upperdir=/var/lib/containers/storage/overlay/1b9026475244313d5b30294d70b79385cc00cac26ffda7478852ca46ee71112e/diff,workdir=/var/lib/containers/storage/overlay/1b9026475244313d5b30294d70b79385cc00cac26ffda7478852ca46ee71112e/work,volatile": using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 125

Comment 2 Alex Jia 2022-03-08 02:15:33 UTC

*** Bug 2061602 has been marked as a duplicate of this bug. ***

Comment 3 Alex Jia 2022-03-08 02:16:45 UTC

This bug hasn't been fixed on rhel9-buildah:9.0.0-4 yet.

Comment 4 Jindrich Novy 2022-03-08 12:05:25 UTC

https://issues.redhat.com/browse/TFT-1108 currently blocks update to buildah-1.24.2 and skopeo-1.6.1 for RHEL9 GA.

Comment 5 Alex Jia 2022-03-16 03:24:28 UTC

This bug has been verified on podman-4.0.2-2.el9_0.x86_64 w/ rhel9-buildah:9.0.0-5.

[root@sweetpig-9 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.0 Beta (Plow)

[root@sweetpig-9 ~]# rpm -q podman crun systemd kernel
podman-4.0.2-2.el9_0.x86_64
crun-1.4.2-1.el9.x86_64
systemd-250-4.el9.x86_64
kernel-5.14.0-70.1.1.el9.x86_64

[root@sweetpig-9 ~]# grep cgroup /proc/mounts 
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
[root@sweetpig-9 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.0.0-5
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.0.0-5...
Getting image source signatures
Copying blob 8ee8520e2f0c done  
Copying blob a308807e614a done  
Copying blob c7250d4af681 done  
Copying config 067ae46188 done  
Writing manifest to image destination
Storing signatures
[root@194a415b13d9 /]# rpm -q buildah fuse-overlayfs
buildah-1.24.1-1.el9.x86_64
fuse-overlayfs-1.8.2-1.el9.x86_64
[root@194a415b13d9 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob de9bc33d7337 done  
Copying blob effc4ea612c8 done  
Copying config b1b0a30a72 done  
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@194a415b13d9 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
d36f66b75ad3     *     b1b0a30a7282 registry.access.redhat.com/ub... ubi8-working-container
[root@194a415b13d9 /]# buildah run --isolation=chroot ubi8-working-container ls /
bin  boot  dev	etc  home  lib	lib64  lost+found  media  mnt  opt  proc  root	run  sbin  srv	sys  tmp  usr  var

Comment 6 Tom Sweeney 2022-03-16 12:47:40 UTC

@jnovy another candidate for closing as current release?

Comment 9 Alex Jia 2022-03-21 08:36:37 UTC

This bug is hit again on podman-4.0.2-3.el9.x86_64 w/ rhel9-buildah:9.0.0-6.

Comment 11 Tom Sweeney 2022-03-21 13:50:45 UTC

@jnovy ping, failed qe

Comment 12 Jindrich Novy 2022-03-21 15:14:53 UTC

Alex, there is no other change in podman-4.0.2-3 apart from rebuild with the new golang.

What is the difference from comment #5 package-wise apart from podman?

Comment 13 Alex Jia 2022-03-21 16:12:15 UTC

(In reply to Jindrich Novy from comment #12)
> Alex, there is no other change in podman-4.0.2-3 apart from rebuild with the
> new golang.
> 
> What is the difference from comment #5 package-wise apart from podman?

Only upgrade podman to 4.0.2-3 w/ new system in the beaker (RHEL-9.0.0-20220320.0),
BTW, it works well if using podman-4.0.2-2.el9_0.x86_64 for the same tests.

[root@sweetpig-21 ~]# rpm -q podman crun systemd kernel
podman-4.0.2-3.el9.x86_64           (different from Comment 5)
crun-1.4.2-1.el9.x86_64             
systemd-250-4.el9.x86_64
kernel-5.14.0-70.2.1.el9_0.x86_64   (different from Comment 5)

and others

[root@sweetpig-21 ~]# rpm -q conmon containers-common fuse-overlayfs slirp4netns 
conmon-2.1.0-1.el9.x86_64
containers-common-1-32.el9.x86_64
fuse-overlayfs-1.8.2-1.el9.x86_64
slirp4netns-1.1.12-4.el9.x86_64


[root@sweetpig-21 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.0.0-6
[root@3f6792baa50d /]# rpm -q buildah fuse-overlayfs
buildah-1.24.1-1.el9.x86_64
fuse-overlayfs-1.8.2-1.el9.x86_64
[root@3f6792baa50d /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob c530010fb61c done  
Copying blob 3de00bb8554b done  
Copying config 552ac8ae42 done  
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@3f6792baa50d /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
9d26efa2157b     *     552ac8ae4291 registry.access.redhat.com/ub... ubi8-working-container
[root@3f6792baa50d /]# buildah run --isolation=chroot ubi8-working-container ls /
ERRO[0000] Unmounting /var/lib/containers/storage/overlay/358b3dfb67b74fa8ad9413ac037dc9d09e93666b2e070658fe92fc65be85a8ff/merged: invalid argument 
error mounting container "9d26efa2157b329d078e4d8d9eb2f469672213273159d81a383d6e33c25e5e5a": error mounting build container "9d26efa2157b329d078e4d8d9eb2f469672213273159d81a383d6e33c25e5e5a": creating overlay mount to /var/lib/containers/storage/overlay/358b3dfb67b74fa8ad9413ac037dc9d09e93666b2e070658fe92fc65be85a8ff/merged, mount_data="nodev,lowerdir=/var/lib/containers/storage/overlay/l/WKLZGWUR54RNQWTRPGSV5VTFKO:/var/lib/containers/storage/overlay/l/ZFM4LF2AAXDJZ6N3752SIYVNQ5,upperdir=/var/lib/containers/storage/overlay/358b3dfb67b74fa8ad9413ac037dc9d09e93666b2e070658fe92fc65be85a8ff/diff,workdir=/var/lib/containers/storage/overlay/358b3dfb67b74fa8ad9413ac037dc9d09e93666b2e070658fe92fc65be85a8ff/work,volatile": using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 125

Comment 15 Alex Jia 2022-04-02 03:02:36 UTC

This bug has been verified on podman-4.0.3-1.el9 w/ rhel9-buildah:9.0.0-8.

[root@kvm-08-guest08 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.0 Beta (Plow)

[root@kvm-08-guest08 ~]# rpm -q podman crun systemd kernel
podman-4.0.3-1.el9.x86_64
crun-1.4.2-1.el9.x86_64
systemd-250-4.el9.x86_64
kernel-5.14.0-70.5.1.el9_0.x86_64

[root@kvm-08-guest08 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.0.0-8
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.0.0-8...
Getting image source signatures
Copying blob ae4eddbce769 done  
Copying blob f0d9a1a0a611 done  
Copying blob d89ed0dbe0c5 done  
Copying config 1b16beb8ed done  
Writing manifest to image destination
Storing signatures
[root@21274d430979 /]# rpm -q buildah fuse-overlayfs
buildah-1.24.2-4.el9_0.x86_64
fuse-overlayfs-1.8.2-1.el9.x86_64
[root@21274d430979 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4eef1fa1f1c1 done  
Copying blob eb24191cef20 done  
Copying config c54243b588 done  
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@21274d430979 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
e2beff80e100     *     c54243b58814 registry.access.redhat.com/ub... ubi8-working-container
[root@21274d430979 /]# buildah run --isolation=chroot ubi8-working-container ls /
bin  boot  dev	etc  home  lib	lib64  lost+found  media  mnt  opt  proc  root	run  sbin  srv	sys  tmp  usr  var
[root@21274d430979 /]# exit
exit
[root@kvm-08-guest08 ~]# echo $?
0

Comment 17 errata-xmlrpc 2022-05-18 02:50:57 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (New rhel9/buildah container image), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4601