Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Additionalimagestores with podman on RHEL8.4/podman 3.2.3 works however,RHEL8.6/podman 4.2 does not work.
Version-Release number of selected component (if applicable):
RHEL 8.6
Podman 4.2
How reproducible:
$ id
uid=1000(test1) gid=1000(test1) groups=1000(test1),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ rpm -q podman
podman-4.2.0-8.module+el8.7.0+17824+66a0202b.x86_64
$ sudo mkdir /var/lib/additionalImageStore
$ sudo chmod -R a+rx /var/lib/additionalImageStore
$ ls -lZd /var/lib/additionalImageStore
drwxr-xr-x. 9 root root unconfined_u:object_r:var_lib_t:s0 169 Mar 20 11:33 /var/lib/additionalImageStore
$ sudo podman --root=/var/lib/additionalImageStore pull registry.access.redhat.com/ubi8
Let's make sure SELinux isn't a factor:
$ sudo setenforce 0
We see the below error only when an additionalImageStore is specified:
$ podman --log-level=debug run -it --storage-opt additionalimagestore=/var/lib/additionalImageStore --name test1 registry.access.redhat.com/ubi8 echo "hello"
[...]
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/conmon: permission denied
DEBU[0000] Received: -1
DEBU[0000] Cleaning up container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/netns-646203c2-9d26-f32e-830c-faacee7c392e for container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227
DEBU[0000] Unmounted container "d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227"
DEBU[0000] ExitCode msg: "runc: time=\"2023-03-20t11:46:57-04:00\" level=warning msg=\"unable to get oom kill count\" error=\"no directory specified for memory.oom_control\"\ntime=\"2023-03-20t11:46:57-04:00\" level=error msg=\"container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \\\"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\\\" to rootfs at \\\"/run/.containerenv\\\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied\": oci permission denied"
Error: runc: time="2023-03-20T11:46:57-04:00" level=warning msg="unable to get oom kill count" error="no directory specified for memory.oom_control"
time="2023-03-20T11:46:57-04:00" level=error msg="container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\" to rootfs at \"/run/.containerenv\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied": OCI permission denied
ALSO,
Adding 8.4 image to shared storage:
IMAGES_DIR=/data/display_rec/images
ARCHIVE=/tmp/ubi-8.4.tar.gz
podman --root= $IMAGES_DIR --storage-opt mount_program="/usr/bin/fuse-overlayfs" pull docker-archive:$ARCHIVE
chmod -R a+rX $IMAGES_DIR/
Getting image source signatures
Copying blob 525ed45dbdb1 skipped: already exists
Copying blob 5bc03dec6239 skipped: already exists
Copying config b1e63aaae5 done
Writing manifest to image destination
Storing signatures
b1e63aaae5cffb78e4af9f3a110dbad67e8013ca3de6d09f1ef496d00641e751
>podman --root=/tmp/test_root2 --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt additionalimagestore=/data/display_rec/images --storage-opt mountopt=squash_to_root run -d --name test --userns=keep-id --security-opt label=disable b1e63aaae5cf /sbin/init
Error: OCI runtime error: runc: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/tmp": tmpcopyup: failed to copy /tmp to /proc/self/fd/13 (/tmp/runctop3357647393/runctmpdir1069372518): open /proc/self/fd/13/: operation not supported
The same command DOES work with root (minus userns keep-id), just not rootless.
Actual results:
Configuration of additionalimagestores for RHEL8.6/podman 4.2 does not work.
Expected results
Configuration of additionalimagestores for RHEL8.6/podman 4.2 should work.
Additional info:
Comment 6Giuseppe Scrivano
2023-03-23 20:26:51 UTC
in the last reproducer it is written:
IMAGES_DIR=/data/display_rec/images
ARCHIVE=/tmp/ubi-8.4.tar.gz
What do they refer to? I can guess, but it is hard to know for sure, can you please indicate how these were created?
The "operation not supported" error smells like the user. extended attribute on tmpfs. Do you see the same error if your root points to another directory that is not on tmpfs?
Can you please share the file /tmp/strace.log that you get running the following command?
podman unshare strace -Z -f -s 1000 -o /tmp/strace.log podman --root=/tmp/test_root2 --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt additionalimagestore=/data/display_rec/images --storage-opt mountopt=squash_to_root run -d --name test --userns=keep-id --security-opt label=disable b1e63aaae5cf /sbin/init
Comment 7Giuseppe Scrivano
2023-03-23 21:45:32 UTC
that seems caused by a new feature in fuse-overlayfs: https://github.com/containers/fuse-overlayfs/issues/304
I'd suggest adding noacl to the mount options: --storage-opt mountopt=noacl
I'll look whether I can do this automatically.
Comment 8Giuseppe Scrivano
2023-03-23 22:03:34 UTC
@jnovy It looks to me like we have fuse-overlayfs v1.8.2-1 in RHEL 8.6, and the problem is there going onward. I don't know which version is on RHEL 8.7/9.1, was that v1.10? Also, which version is in 8.8/9.2, 1.10 at this point? My thinking is we do a ZeroDay for RHEL 8.8/9.2 using fuse-overlayfs v1.11 that Giuseppe put together. Does that work? If so, I'll spin up some BZs to submit against.
Comment 32Giuseppe Scrivano
2023-04-17 15:40:38 UTC
The customer has already verified the fix.
The one you are encountering seems like a different error.
Have you chowned the /tmp/addi/overlay-images store to 755 before?
Comment 38Giuseppe Scrivano
2023-04-20 07:46:24 UTC
You need to specify the mount program, otherwise Podman will use native overlay that doesn't support running from an NFS store.
Can you try with the following command:
podman run --storage-opt mount_program="/usr/bin/fuse-overlayfs" --storage-opt additionalimagestore=/var/lib/additionalImageStore --rm -it --name test1 registry.access.redhat.com/ubi8 echo "hello"
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: container-tools:rhel8 security and bug fix update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2023:6939
Description of problem: Additionalimagestores with podman on RHEL8.4/podman 3.2.3 works however,RHEL8.6/podman 4.2 does not work. Version-Release number of selected component (if applicable): RHEL 8.6 Podman 4.2 How reproducible: $ id uid=1000(test1) gid=1000(test1) groups=1000(test1),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 $ rpm -q podman podman-4.2.0-8.module+el8.7.0+17824+66a0202b.x86_64 $ sudo mkdir /var/lib/additionalImageStore $ sudo chmod -R a+rx /var/lib/additionalImageStore $ ls -lZd /var/lib/additionalImageStore drwxr-xr-x. 9 root root unconfined_u:object_r:var_lib_t:s0 169 Mar 20 11:33 /var/lib/additionalImageStore $ sudo podman --root=/var/lib/additionalImageStore pull registry.access.redhat.com/ubi8 Let's make sure SELinux isn't a factor: $ sudo setenforce 0 We see the below error only when an additionalImageStore is specified: $ podman --log-level=debug run -it --storage-opt additionalimagestore=/var/lib/additionalImageStore --name test1 registry.access.redhat.com/ubi8 echo "hello" [...] INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/conmon: permission denied DEBU[0000] Received: -1 DEBU[0000] Cleaning up container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227 DEBU[0000] Tearing down network namespace at /run/user/1000/netns/netns-646203c2-9d26-f32e-830c-faacee7c392e for container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227 DEBU[0000] Unmounted container "d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227" DEBU[0000] ExitCode msg: "runc: time=\"2023-03-20t11:46:57-04:00\" level=warning msg=\"unable to get oom kill count\" error=\"no directory specified for memory.oom_control\"\ntime=\"2023-03-20t11:46:57-04:00\" level=error msg=\"container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \\\"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\\\" to rootfs at \\\"/run/.containerenv\\\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied\": oci permission denied" Error: runc: time="2023-03-20T11:46:57-04:00" level=warning msg="unable to get oom kill count" error="no directory specified for memory.oom_control" time="2023-03-20T11:46:57-04:00" level=error msg="container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\" to rootfs at \"/run/.containerenv\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied": OCI permission denied ALSO, Adding 8.4 image to shared storage: IMAGES_DIR=/data/display_rec/images ARCHIVE=/tmp/ubi-8.4.tar.gz podman --root= $IMAGES_DIR --storage-opt mount_program="/usr/bin/fuse-overlayfs" pull docker-archive:$ARCHIVE chmod -R a+rX $IMAGES_DIR/ Getting image source signatures Copying blob 525ed45dbdb1 skipped: already exists Copying blob 5bc03dec6239 skipped: already exists Copying config b1e63aaae5 done Writing manifest to image destination Storing signatures b1e63aaae5cffb78e4af9f3a110dbad67e8013ca3de6d09f1ef496d00641e751 >podman --root=/tmp/test_root2 --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt additionalimagestore=/data/display_rec/images --storage-opt mountopt=squash_to_root run -d --name test --userns=keep-id --security-opt label=disable b1e63aaae5cf /sbin/init Error: OCI runtime error: runc: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/tmp": tmpcopyup: failed to copy /tmp to /proc/self/fd/13 (/tmp/runctop3357647393/runctmpdir1069372518): open /proc/self/fd/13/: operation not supported The same command DOES work with root (minus userns keep-id), just not rootless. Actual results: Configuration of additionalimagestores for RHEL8.6/podman 4.2 does not work. Expected results Configuration of additionalimagestores for RHEL8.6/podman 4.2 should work. Additional info: