RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2180118 - Rootless podman with additionalimagestore does not work on RHEL8.6
Summary: Rootless podman with additionalimagestore does not work on RHEL8.6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: fuse-overlayfs
Version: 8.6
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Yuhui Jiang
URL:
Whiteboard:
Depends On:
Blocks: 2185115 2185132
TreeView+ depends on / blocked
 
Reported: 2023-03-20 18:26 UTC by Carroline
Modified: 2023-11-14 16:38 UTC (History)
18 users (show)

Fixed In Version: fuse-overlayfs-1.11-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2185115 2185132 (view as bug list)
Environment:
Last Closed: 2023-11-14 15:29:00 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers fuse-overlayfs pull 389 0 None Merged main: check that ACLs are supported in the lower layers 2023-03-24 15:50:14 UTC
Red Hat Issue Tracker ACCELFIX-27 0 None None None 2023-04-06 23:46:53 UTC
Red Hat Issue Tracker RHELPLAN-152453 0 None None None 2023-03-20 18:28:56 UTC

Description Carroline 2023-03-20 18:26:43 UTC 
Description of problem:
Additionalimagestores with podman on RHEL8.4/podman 3.2.3 works however,RHEL8.6/podman 4.2 does not work.


Version-Release number of selected component (if applicable):
RHEL 8.6
Podman 4.2

How reproducible:

$ id
uid=1000(test1) gid=1000(test1) groups=1000(test1),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

$ rpm -q podman
podman-4.2.0-8.module+el8.7.0+17824+66a0202b.x86_64

$ sudo mkdir /var/lib/additionalImageStore

$ sudo chmod -R a+rx /var/lib/additionalImageStore

$ ls -lZd /var/lib/additionalImageStore
drwxr-xr-x. 9 root root unconfined_u:object_r:var_lib_t:s0 169 Mar 20 11:33 /var/lib/additionalImageStore

$ sudo podman --root=/var/lib/additionalImageStore pull registry.access.redhat.com/ubi8

Let's make sure SELinux isn't a factor:

$ sudo setenforce 0

We see the below error only when an additionalImageStore is specified:

$ podman --log-level=debug run -it --storage-opt additionalimagestore=/var/lib/additionalImageStore --name test1 registry.access.redhat.com/ubi8 echo "hello"
[...]
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/conmon: permission denied
DEBU[0000] Received: -1
DEBU[0000] Cleaning up container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/netns-646203c2-9d26-f32e-830c-faacee7c392e for container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227
DEBU[0000] Unmounted container "d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227"
DEBU[0000] ExitCode msg: "runc: time=\"2023-03-20t11:46:57-04:00\" level=warning msg=\"unable to get oom kill count\" error=\"no directory specified for memory.oom_control\"\ntime=\"2023-03-20t11:46:57-04:00\" level=error msg=\"container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \\\"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\\\" to rootfs at \\\"/run/.containerenv\\\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied\": oci permission denied"
Error: runc: time="2023-03-20T11:46:57-04:00" level=warning msg="unable to get oom kill count" error="no directory specified for memory.oom_control"
time="2023-03-20T11:46:57-04:00" level=error msg="container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\" to rootfs at \"/run/.containerenv\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied": OCI permission denied

ALSO,


Adding 8.4 image to shared storage:

IMAGES_DIR=/data/display_rec/images
ARCHIVE=/tmp/ubi-8.4.tar.gz

podman --root= $IMAGES_DIR --storage-opt mount_program="/usr/bin/fuse-overlayfs" pull docker-archive:$ARCHIVE
chmod -R a+rX $IMAGES_DIR/

Getting image source signatures
Copying blob 525ed45dbdb1 skipped: already exists
Copying blob 5bc03dec6239 skipped: already exists
Copying config b1e63aaae5 done
Writing manifest to image destination
Storing signatures
b1e63aaae5cffb78e4af9f3a110dbad67e8013ca3de6d09f1ef496d00641e751

>podman --root=/tmp/test_root2 --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt additionalimagestore=/data/display_rec/images --storage-opt mountopt=squash_to_root run -d --name test --userns=keep-id --security-opt label=disable b1e63aaae5cf /sbin/init
Error: OCI runtime error: runc: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/tmp": tmpcopyup: failed to copy /tmp to /proc/self/fd/13 (/tmp/runctop3357647393/runctmpdir1069372518): open /proc/self/fd/13/: operation not supported

The same command DOES work with root (minus userns keep-id), just not rootless.

 

Actual results:
Configuration of additionalimagestores for RHEL8.6/podman 4.2 does not work.


Expected results
Configuration of additionalimagestores for RHEL8.6/podman 4.2 should work.


Additional info:
Comment 6 Giuseppe Scrivano 2023-03-23 20:26:51 UTC 
in the last reproducer it is written:

IMAGES_DIR=/data/display_rec/images
ARCHIVE=/tmp/ubi-8.4.tar.gz

What do they refer to?  I can guess, but it is hard to know for sure, can you please indicate how these were created?

The "operation not supported" error smells like the user. extended attribute on tmpfs.  Do you see the same error if your root points to another directory that is not on tmpfs?

Can you please share the file /tmp/strace.log that you get running the following command?

podman unshare strace -Z -f -s 1000 -o /tmp/strace.log podman --root=/tmp/test_root2 --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt additionalimagestore=/data/display_rec/images --storage-opt mountopt=squash_to_root run -d --name test --userns=keep-id --security-opt label=disable b1e63aaae5cf /sbin/init
Comment 7 Giuseppe Scrivano 2023-03-23 21:45:32 UTC 
that seems caused by a new feature in fuse-overlayfs: https://github.com/containers/fuse-overlayfs/issues/304

I'd suggest adding noacl to the mount options: --storage-opt mountopt=noacl

I'll look whether I can do this automatically.
Comment 8 Giuseppe Scrivano 2023-03-23 22:03:34 UTC 
PR for fuse-overlayfs: https://github.com/containers/fuse-overlayfs/pull/389
Comment 17 Giuseppe Scrivano 2023-03-25 10:34:07 UTC 
I've cut fuse-overlayfs v1.11 with the fix above
Comment 20 Giuseppe Scrivano 2023-04-04 09:10:18 UTC 
it is fixed in fuse-overlayfs v1.11
Comment 21 Tom Sweeney 2023-04-04 20:40:42 UTC 
@jnovy It looks to me like we have fuse-overlayfs v1.8.2-1 in RHEL 8.6, and the problem is there going onward.  I don't know which version is on RHEL 8.7/9.1, was that v1.10?  Also, which version is in 8.8/9.2, 1.10 at this point?  My thinking is we do a ZeroDay for RHEL 8.8/9.2 using fuse-overlayfs v1.11 that Giuseppe put together. Does that work?  If so, I'll spin up some BZs to submit against.
Comment 32 Giuseppe Scrivano 2023-04-17 15:40:38 UTC 
The customer has already verified the fix.

The one you are encountering seems like a different error.

Have you chowned the /tmp/addi/overlay-images store to 755 before?
Comment 38 Giuseppe Scrivano 2023-04-20 07:46:24 UTC 
You need to specify the mount program, otherwise Podman will use native overlay that doesn't support running from an NFS store.

Can you try with the following command:

podman run --storage-opt mount_program="/usr/bin/fuse-overlayfs" --storage-opt additionalimagestore=/var/lib/additionalImageStore --rm -it --name test1 registry.access.redhat.com/ubi8 echo "hello"
Comment 51 errata-xmlrpc 2023-11-14 15:29:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6939
Note You need to log in before you can comment on or make changes to this bug.