Bug 2180118
| Summary: | Rootless podman with additionalimagestore does not work on RHEL8.6 | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Carroline <cpippin> | |
| Component: | fuse-overlayfs | Assignee: | Jindrich Novy <jnovy> | |
| Status: | CLOSED ERRATA | QA Contact: | Yuhui Jiang <yujiang> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 8.6 | CC: | bbaude, dornelas, dwalsh, gscrivan, jligon, jnovy, julian.n.manfredi, kemyers, lsm5, mboddu, mheon, nhenders, pthomas, tsweeney, umohnani, wwurzbac, ypu, yujiang | |
| Target Milestone: | rc | Keywords: | Triaged, ZStream | |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | fuse-overlayfs-1.11-1.el8 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2185115 2185132 (view as bug list) | Environment: | ||
| Last Closed: | 2023-11-14 15:29:00 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2185115, 2185132 | |||
in the last reproducer it is written: IMAGES_DIR=/data/display_rec/images ARCHIVE=/tmp/ubi-8.4.tar.gz What do they refer to? I can guess, but it is hard to know for sure, can you please indicate how these were created? The "operation not supported" error smells like the user. extended attribute on tmpfs. Do you see the same error if your root points to another directory that is not on tmpfs? Can you please share the file /tmp/strace.log that you get running the following command? podman unshare strace -Z -f -s 1000 -o /tmp/strace.log podman --root=/tmp/test_root2 --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt additionalimagestore=/data/display_rec/images --storage-opt mountopt=squash_to_root run -d --name test --userns=keep-id --security-opt label=disable b1e63aaae5cf /sbin/init that seems caused by a new feature in fuse-overlayfs: https://github.com/containers/fuse-overlayfs/issues/304 I'd suggest adding noacl to the mount options: --storage-opt mountopt=noacl I'll look whether I can do this automatically. PR for fuse-overlayfs: https://github.com/containers/fuse-overlayfs/pull/389 I've cut fuse-overlayfs v1.11 with the fix above it is fixed in fuse-overlayfs v1.11 @jnovy It looks to me like we have fuse-overlayfs v1.8.2-1 in RHEL 8.6, and the problem is there going onward. I don't know which version is on RHEL 8.7/9.1, was that v1.10? Also, which version is in 8.8/9.2, 1.10 at this point? My thinking is we do a ZeroDay for RHEL 8.8/9.2 using fuse-overlayfs v1.11 that Giuseppe put together. Does that work? If so, I'll spin up some BZs to submit against. The customer has already verified the fix. The one you are encountering seems like a different error. Have you chowned the /tmp/addi/overlay-images store to 755 before? You need to specify the mount program, otherwise Podman will use native overlay that doesn't support running from an NFS store. Can you try with the following command: podman run --storage-opt mount_program="/usr/bin/fuse-overlayfs" --storage-opt additionalimagestore=/var/lib/additionalImageStore --rm -it --name test1 registry.access.redhat.com/ubi8 echo "hello" Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: container-tools:rhel8 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6939 |
Description of problem: Additionalimagestores with podman on RHEL8.4/podman 3.2.3 works however,RHEL8.6/podman 4.2 does not work. Version-Release number of selected component (if applicable): RHEL 8.6 Podman 4.2 How reproducible: $ id uid=1000(test1) gid=1000(test1) groups=1000(test1),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 $ rpm -q podman podman-4.2.0-8.module+el8.7.0+17824+66a0202b.x86_64 $ sudo mkdir /var/lib/additionalImageStore $ sudo chmod -R a+rx /var/lib/additionalImageStore $ ls -lZd /var/lib/additionalImageStore drwxr-xr-x. 9 root root unconfined_u:object_r:var_lib_t:s0 169 Mar 20 11:33 /var/lib/additionalImageStore $ sudo podman --root=/var/lib/additionalImageStore pull registry.access.redhat.com/ubi8 Let's make sure SELinux isn't a factor: $ sudo setenforce 0 We see the below error only when an additionalImageStore is specified: $ podman --log-level=debug run -it --storage-opt additionalimagestore=/var/lib/additionalImageStore --name test1 registry.access.redhat.com/ubi8 echo "hello" [...] INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/conmon: permission denied DEBU[0000] Received: -1 DEBU[0000] Cleaning up container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227 DEBU[0000] Tearing down network namespace at /run/user/1000/netns/netns-646203c2-9d26-f32e-830c-faacee7c392e for container d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227 DEBU[0000] Unmounted container "d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227" DEBU[0000] ExitCode msg: "runc: time=\"2023-03-20t11:46:57-04:00\" level=warning msg=\"unable to get oom kill count\" error=\"no directory specified for memory.oom_control\"\ntime=\"2023-03-20t11:46:57-04:00\" level=error msg=\"container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \\\"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\\\" to rootfs at \\\"/run/.containerenv\\\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied\": oci permission denied" Error: runc: time="2023-03-20T11:46:57-04:00" level=warning msg="unable to get oom kill count" error="no directory specified for memory.oom_control" time="2023-03-20T11:46:57-04:00" level=error msg="container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \"/run/user/1000/containers/overlay-containers/d21f60fe14ea87fa59b871dd7518b51d286dc1478db433c19ddf5196f5871227/userdata/.containerenv\" to rootfs at \"/run/.containerenv\" caused: open /home/test1/.local/share/containers/storage/overlay/62de0aa02ab45fe58532a2c1dd19e33861cd9e6a030c130a13fe6b59892fa610/merged/run/.containerenv: permission denied": OCI permission denied ALSO, Adding 8.4 image to shared storage: IMAGES_DIR=/data/display_rec/images ARCHIVE=/tmp/ubi-8.4.tar.gz podman --root= $IMAGES_DIR --storage-opt mount_program="/usr/bin/fuse-overlayfs" pull docker-archive:$ARCHIVE chmod -R a+rX $IMAGES_DIR/ Getting image source signatures Copying blob 525ed45dbdb1 skipped: already exists Copying blob 5bc03dec6239 skipped: already exists Copying config b1e63aaae5 done Writing manifest to image destination Storing signatures b1e63aaae5cffb78e4af9f3a110dbad67e8013ca3de6d09f1ef496d00641e751 >podman --root=/tmp/test_root2 --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt additionalimagestore=/data/display_rec/images --storage-opt mountopt=squash_to_root run -d --name test --userns=keep-id --security-opt label=disable b1e63aaae5cf /sbin/init Error: OCI runtime error: runc: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/tmp": tmpcopyup: failed to copy /tmp to /proc/self/fd/13 (/tmp/runctop3357647393/runctmpdir1069372518): open /proc/self/fd/13/: operation not supported The same command DOES work with root (minus userns keep-id), just not rootless. Actual results: Configuration of additionalimagestores for RHEL8.6/podman 4.2 does not work. Expected results Configuration of additionalimagestores for RHEL8.6/podman 4.2 should work. Additional info: