Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2057243

Summary:	failed to open /dev/fuse: Operation not permitted
Product:	Red Hat Enterprise Linux 8	Reporter:	Alex Jia <ajia>
Component:	buildah-container	Assignee:	Jindrich Novy <jnovy>
Status:	CLOSED ERRATA	QA Contact:	Alex Jia <ajia>
Severity:	high	Docs Contact:	Michelle Bearer <mbearer>
Priority:	unspecified
Version:	8.6	CC:	dwalsh, jnovy, tsweeney, ypu
Target Milestone:	rc	Keywords:	Reopened
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	rhel8-buildah:8.6-4	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2057337 2066138 (view as bug list)		Environment:
Last Closed:	2022-05-10 21:27:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2044787, 2057337

Description Alex Jia 2022-02-23 03:30:45 UTC

Description of problem:
Failed to run embedded container inside the rhel8-buildah:8.6-2 and got error like this "xxx using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted xxx".

Version-Release number of selected component (if applicable):
[root@hpe-dl380pgen8-02-vm-9 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)
[root@hpe-dl380pgen8-02-vm-9 ~]# rpm -q podman runc systemd kernel
podman-4.0.0-0.29.module+el8.6.0+14295+adafbc4c.x86_64
runc-1.0.3-1.module+el8.6.0+14295+adafbc4c.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-367.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-2
2. buildah from registry.access.redhat.com/ubi8
3. buildah run --isolation=chroot ubi8-working-container ls 

Actual results:
[root@hpe-dl380pgen8-02-vm-9 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-2
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-2...
Getting image source signatures
Copying blob 28ced426e3f1 done
Copying blob 0b6f949925e2 done
Copying blob e79d3f31aa4c done
Copying config 3bfe906ec6 done
Writing manifest to image destination
Storing signatures
[root@019326d5da82 /]# rpm -q buildah fuse-overlayfs
buildah-1.23.1-3.module+el8.6.0+13412+2981c13c.x86_64
fuse-overlayfs-1.8-1.module+el8.6.0+13730+5dcf3a04.x86_64
[root@019326d5da82 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 5dcbdc60ea6b done
Copying blob 8671113e1c57 done
Copying config b81e86a2cb done
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@019326d5da82 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
1132ea4bdd51     *     b81e86a2cb9a registry.access.redhat.com/ub... ubi8-working-container
[root@019326d5da82 /]# buildah run --isolation=chroot ubi8-working-container ls /
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/merged: invalid argument
error mounting container "1132ea4bdd51b49feb3450f4bada455ff41cf582cfc7f4fa14e6c3fffa40280f": error mounting build container "1132ea4bdd51b49feb3450f4bada455ff41cf582cfc7f4fa14e6c3fffa40280f": error creating overlay mount to /var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/R6BKACDZ4WVQTYCL7JE3ISB3HT:/var/lib/containers/storage/overlay/l/273NNQHV56HG4M2MXQARYMRFY5,upperdir=/var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/diff,workdir=/var/lib/containers/storage/overlay/1d15df68d22ede6dcb3b869079829db82e42388110a37bf3b2d04ea0c594adb3/work,volatile": using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 125

Expected results:
fix it.

Additional info:

Comment 1 Tom Sweeney 2022-02-23 14:13:42 UTC

Jindrich, I think this might be a packaging issue.  If not, please let me know and I'll assign elsewhere.

Comment 2 Alex Jia 2022-03-11 12:47:02 UTC

[root@sweetpig-22 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-4
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-4...
Getting image source signatures
Copying blob 968ce03aa7c4 done
Copying blob 6bc6fc7fa3c2 done
Copying blob 2208f6c75e6b done
Copying config 68ff495c39 done
Writing manifest to image destination
Storing signatures
[root@d61c4e24eed6 /]# rpm -q buildah fuse-overlayfs
buildah-1.24.2-2.module+el8.6.0+14379+4ec2a99a.x86_64
fuse-overlayfs-1.8.2-1.module+el8.6.0+14379+4ec2a99a.x86_64
[root@d61c4e24eed6 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 8dfe9326f733 done
Copying blob 0d875a68bf99 done
Copying config 52de04277b done
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@d61c4e24eed6 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
dddc404ce84b     *     52de04277b39 registry.access.redhat.com/ub... ubi8-working-container
[root@d61c4e24eed6 /]# buildah run --isolation=chroot ubi8-working-container ls /
ERRO[0000] Unmounting /var/lib/containers/storage/overlay/97d65d3c6ce99f129de3a53da1bf0925055cdb61103afd41daec1e35b636900d/merged: invalid argument
error mounting container "dddc404ce84bfacef095a1435a05b4ca62408e3e1ebd24273cd81ba9a50b387c": error mounting build container "dddc404ce84bfacef095a1435a05b4ca62408e3e1ebd24273cd81ba9a50b387c": creating overlay mount to /var/lib/containers/storage/overlay/97d65d3c6ce99f129de3a53da1bf0925055cdb61103afd41daec1e35b636900d/merged, mount_data="nodev,lowerdir=/var/lib/containers/storage/overlay/l/HE3EXTDYFV5L2WHPDHZXOAOAGD:/var/lib/containers/storage/overlay/l/QSOSCSRTNBCD5MC4KVC552A3CX,upperdir=/var/lib/containers/storage/overlay/97d65d3c6ce99f129de3a53da1bf0925055cdb61103afd41daec1e35b636900d/diff,workdir=/var/lib/containers/storage/overlay/97d65d3c6ce99f129de3a53da1bf0925055cdb61103afd41daec1e35b636900d/work,volatile": using mount program /usr/bin/fuse-overlayfs: fuse: failed to open /dev/fuse: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 125

Comment 3 Alex Jia 2022-03-14 16:19:49 UTC

The podman v3.4.5 is okay, but it didn't work for podman v4.0 + rhel8-buildah:8.6-4.

Comment 4 Jindrich Novy 2022-03-14 16:59:12 UTC

Alex, do you mind re-checking this with podman-4.0.2? The advisory was updated today. Thanks.

Comment 5 Alex Jia 2022-03-16 02:58:32 UTC

(In reply to Jindrich Novy from comment #4)
> Alex, do you mind re-checking this with podman-4.0.2? The advisory was
> updated today. Thanks.

it works well for podman-4.0.2-2.module+el8.6.0+14478+dcf60c19 w/ rhel8-buildah:8.6-4.

[root@sweetpig-7 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)

[root@sweetpig-7 ~]# rpm -q podman runc systemd kernel
podman-4.0.2-2.module+el8.6.0+14478+dcf60c19.x86_64
runc-1.0.3-1.module+el8.6.0+14378+7b18e39f.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-372.el8.x86_64

[root@sweetpig-7 ~]# grep cgroup /proc/mounts 
tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,size=7535720k,nr_inodes=1883930,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0

[root@sweetpig-7 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-4
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-4...
Getting image source signatures
Copying blob 968ce03aa7c4 done  
Copying blob 6bc6fc7fa3c2 done  
Copying blob 2208f6c75e6b done  
Copying config 68ff495c39 done  
Writing manifest to image destination
Storing signatures
[root@7d7501e67645 /]# rpm -q buildah fuse-overlayfs
buildah-1.24.2-2.module+el8.6.0+14379+4ec2a99a.x86_64
fuse-overlayfs-1.8.2-1.module+el8.6.0+14379+4ec2a99a.x86_64
[root@7d7501e67645 /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob effc4ea612c8 done  
Copying blob de9bc33d7337 done  
Copying config b1b0a30a72 done  
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@7d7501e67645 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
a8ee1eeda649     *     b1b0a30a7282 registry.access.redhat.com/ub... ubi8-working-container
[root@7d7501e67645 /]# buildah run --isolation=chroot ubi8-working-container ls /
bin  boot  dev	etc  home  lib	lib64  lost+found  media  mnt  opt  proc  root	run  sbin  srv	sys  tmp  usr  var

Comment 6 Tom Sweeney 2022-03-16 12:45:50 UTC

@jnovy As Alex verified this with the latest, can we close this as current release?

Comment 10 Alex Jia 2022-03-21 02:56:24 UTC

[root@sweetpig-20 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)

[root@sweetpig-20 ~]# rpm -q podman runc systemd kernel
podman-4.0.2-2.module+el8.6.0+14488+6524fb7f.x86_64
runc-1.0.3-2.module+el8.6.0+14488+6524fb7f.x86_64
systemd-239-58.el8.x86_64
kernel-4.18.0-372.2.1.el8.x86_64

[root@sweetpig-20 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-5
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.6-5...
Getting image source signatures
Copying blob a6fd35289163 done
Copying blob 2907092333e1 done
Copying blob 76ea87ceff0d done
Copying config eda4e8afab done
Writing manifest to image destination
Storing signatures
[root@354215393fce /]# rpm -q buildah fuse-overlayfs
buildah-1.24.2-2.module+el8.6.0+14488+6524fb7f.x86_64
fuse-overlayfs-1.8.2-1.module+el8.6.0+14488+6524fb7f.x86_64
[root@354215393fce /]# buildah from registry.access.redhat.com/ubi8
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 3de00bb8554b done
Copying blob c530010fb61c done
Copying config 552ac8ae42 done
Writing manifest to image destination
Storing signatures
ubi8-working-container
[root@354215393fce /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
8f0456e15e08     *     552ac8ae4291 registry.access.redhat.com/ub... ubi8-working-container
[root@354215393fce /]# buildah run  --isolation=chroot ubi8-working-container ls
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Comment 11 Alex Jia 2022-03-24 11:55:40 UTC

To close this bug per Comment 10.

Comment 13 errata-xmlrpc 2022-05-10 21:27:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rhel8/buildah container image update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2157