RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2166225 - Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted
Summary: Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: buildah
Version: 9.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Gabriela Nečasová
QA Contact: Alex Jia
URL:
Whiteboard:
Depends On: 2166195 2178263
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-02-01 08:23 UTC by Alex Jia
Modified: 2023-04-11 01:18 UTC (History)
12 users (show)

Fixed In Version: buildah-1.29.0-2.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2166195
Environment:
Last Closed: 2023-03-06 19:18:51 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers/storage/commit/1af3928e9bf16d9c39d3d60bf3ec6bb7167989a6 0 None None None 2023-02-07 10:33:29 UTC
Red Hat Issue Tracker RHELPLAN-147242 0 None None None 2023-02-01 08:24:58 UTC

Description Alex Jia 2023-02-01 08:23:06 UTC 
+++ This bug was initially created as a clone of Bug #2166195 +++

Description of problem:
It's failed to run buildah command inside the buildah-container and got error like this "Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted"

Version-Release number of selected component (if applicable):
[root@kvm-04-guest12 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.8 Beta (Ootpa)

[root@kvm-04-guest12 ~]# rpm -q podman runc systemd kernel
podman-4.3.1-2.module+el8.8.0+17695+8a9c0c1b.x86_64
runc-1.1.4-1.module+el8.8.0+17695+8a9c0c1b.x86_64
systemd-239-70.el8.x86_64
kernel-4.18.0-453.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. podman run --rm --device /dev/fuse -it registry.XXX/rhel8-buildah:8.8-1
2. buildah from ubi8

Actual results:
[root@kvm-04-guest12 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel8-buildah:8.8-1
[root@b4036ea892bd /]# rpm -q buildah fuse-overlayfs
buildah-1.28.2-2.module+el8.8.0+17695+8a9c0c1b.x86_64
fuse-overlayfs-1.10-1.module+el8.8.0+17695+8a9c0c1b.x86_64
[root@b4036ea892bd /]# buildah --log-level=debug from ubi8
DEBU[0000] Pull Policy for pull [ifnewer]               
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
WARN[0000] failed to shutdown storage: "mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted" 


Expected results:


Additional info:

--- Additional comment from Alex Jia on 2023-02-01 08:16:13 UTC ---

(In reply to Alex Jia from comment #0)
> Description of problem:
> It's failed to pull image inside the buildah-container and got error like
> this "Error: mount
> /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay,
> flags: 0x1000: operation not permitted"
> 

In fact, it's failed to run any buildah comand except help inside builldah container.

[root@b4036ea892bd /]# buildah --log-level=debug info
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
WARN[0000] failed to shutdown storage: "mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted"
Comment 1 Alex Jia 2023-02-01 08:24:56 UTC 
[root@kvm-04-guest14 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.2 Beta (Plow)

[root@kvm-04-guest14 ~]# rpm -q podman crun systemd kernel
podman-4.3.1-3.el9.x86_64
crun-1.7.2-2.el9.x86_64
systemd-252-3.el9.x86_64
kernel-5.14.0-247.el9.x86_64

[root@kvm-04-guest14 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-1
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-1...
Getting image source signatures
Copying blob 612007f8c61d done  
Copying blob 84d5648acfd5 done  
Copying config 741abe95c7 done  
Writing manifest to image destination
Storing signatures
[root@8d95b772061e /]# rpm -q buildah fuse-overlayfs
buildah-1.28.0-2.el9.x86_64
fuse-overlayfs-1.10-1.el9.x86_64
[root@8d95b772061e /]# buildah --log-level=debug info
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: imagestore=/var/lib/shared          
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
WARN[0000] failed to shutdown storage: "mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted"
Comment 6 Giuseppe Scrivano 2023-02-14 10:15:41 UTC 
that seems a regression caused by https://github.com/containers/common/commit/f39f2a3f8c7680b9e456b9d235570e511807d6c6 that drops sys_chroot from the default capabilities.

Dan, should we add the capability back or is the expectation to use --cap-add sys_chroot for build containers?
Comment 7 Daniel Walsh 2023-02-15 07:38:42 UTC 
Why does a mount need CAP_SYS_CHROOT?  This seems to be working fine in Fedora, why is this broken just in RHEL?
Comment 8 Daniel Walsh 2023-02-15 07:40:49 UTC 
Ok so this is buildah --isolation chroot within a container.

Is this a regression in RHEL9?  I don't think so, I thought we defaulted to no CHROOT at the release of RHEL9?
Comment 9 Giuseppe Scrivano 2023-02-16 08:40:48 UTC 
the commit is quite recent: Nov 22, 2022, so it might not be in older releases.

Should we consider it a breaking change?

If not, I think it is safer to not add CAP_SYS_CHROOT by default to all containers, since it is needed in very few cases (like buildah --isolation chroot).  Would it be enough to document the requirement of using  --cap-add sys_chroot?
Comment 10 Daniel Walsh 2023-02-16 08:45:49 UTC 
Yes I think we should improve the documentation. And recommend users do --cap-add sys_chroot.
Comment 11 Giuseppe Scrivano 2023-02-16 10:51:34 UTC 
Alex, could you please try adding --cap-add sys_chroot?
Comment 12 Alex Jia 2023-02-16 11:14:18 UTC 
(In reply to Giuseppe Scrivano from comment #11)
> Alex, could you please try adding --cap-add sys_chroot?

It works well for me with buildah-1.29.0-2.el9 and buildah-1.29.0-3.el9.

[root@kvm-04-guest15 ~]# podman run --cap-add sys_chroot --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-1
...ignore...
[root@ae3bc3f4dc57 /]# rpm -Uvh buildah-1.29.0-3.el9.x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:buildah-1:1.29.0-3.el9           ################################# [ 50%]
Cleaning up / removing...
   2:buildah-1:1.29.0-2.el9           ################################# [100%]
[root@ae3bc3f4dc57 /]# buildah from ubi9
ubi9-working-container
[root@ae3bc3f4dc57 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
37e25412c686     *     10acc174412e registry.access.redhat.com/ub... ubi9-working-container
[root@ae3bc3f4dc57 /]# buildah run --isolation=chroot ubi9-working-container ls /
afs  bin  boot	dev  etc  home	lib  lib64  lost+found	media  mnt  opt  proc  root  run  sbin	srv  sys  tmp  usr  var
Comment 13 Tom Sweeney 2023-02-16 23:32:55 UTC 
@dwalsh so is this only a change documentation BZ at this point?  If so, where do we document this.

@gnecasov heads up.
Comment 15 Daniel Walsh 2023-02-28 14:06:39 UTC 
We should list the current list of default capabilities.

I think Red Hat could add a new article explaining the case where a container need sys_chroot.
Comment 16 Tom Sweeney 2023-02-28 20:21:24 UTC 
I'm assigning this to @gnecasov for the documentation additions.  Gabi, please contact Giuseppe and Dan if you have questions.
Comment 19 Tom Sweeney 2023-03-06 19:18:51 UTC 
Gabi and Giuseppe, thanks a bunch for working through this.  I'm going to close this current release as the documentation has been published with the expected behavior outlined..
Comment 20 Alex Jia 2023-04-11 01:18:21 UTC 
Just a record.

[root@kvm-01-guest10 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.2 (Plow)

[root@kvm-01-guest10 ~]# rpm -q buildah podman containers-common kernel
buildah-1.29.1-2.el9_2.x86_64
podman-4.4.1-7.el9_2.x86_64
containers-common-1-52.el9_2.x86_64
kernel-5.14.0-284.10.1.el9_2.x86_64


[root@kvm-01-guest10 ~]# podman run --rm --device /dev/fuse -it registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-3
Trying to pull registry-proxy.engineering.redhat.com/rh-osbs/rhel9-buildah:9.2-3...
Getting image source signatures
Copying blob 8fcb66e1c5fd done  
Copying blob f375df5edc3f done  
Copying config 05a32ed22e done  
Writing manifest to image destination
Storing signatures

[root@27d372de3718 /]# rpm -q buildah fuse-overlayfs containers-common
buildah-1.29.0-2.el9.x86_64
fuse-overlayfs-1.10-2.el9.x86_64
containers-common-1-49.el9_1.x86_64
[root@27d372de3718 /]# buildah from ubi9
Resolved "ubi9" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi9:latest...

Getting image source signatures
Checking if image destination supports signatures
Copying blob 2a625e4afab5 done  
Copying config 9877f06ecc done  
Writing manifest to image destination
Storing signatures
ubi9-working-container
[root@27d372de3718 /]# 
[root@27d372de3718 /]# buildah ps
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
728e8a3e8f23     *     9877f06ecc6f registry.access.redhat.com/ub... ubi9-working-container
[root@27d372de3718 /]# buildah run --isolation=chroot ubi9-working-container ls /
afs  bin  boot	dev  etc  home	lib  lib64  lost+found	media  mnt  opt  proc  root  run  sbin	srv  sys  tmp  usr  var
Note You need to log in before you can comment on or make changes to this bug.