moving from apptainer 1.4.1-1.el9 to 1.4.4-1.el9, or more specifically from /usr/libexec/apptainer/bin/fuse2fs 1.47.2 to 1.47.3, seem to have broken the overlay functionality, at least in the way we use it (from debug: "Mount rootfs in read-only mode", "Image type is 4096") at the moment we reverted the fuse2fs binary to the older version
Please give complete instructions on how to reproduce. This change is from upstream so preferably instead report an issue at https://github.com/apptainer/apptainer. There were bugs with read-only and fuse2fs in 1.4.3 that were patched in 1.4.4 so perhaps there are more.
this is an example command apptainer --debug shell --overlay /path/to/overlay.img /path/to/image.sif we will likely report upstream too as there are a couple similar issues already, as you said with "fixed in 1.4.4" thanks
That works for me so more details are needed, such as how do you create the overlay image and sif images. Complete instructions from scratch to reproduce.
here is a full reproduction as we received from the internal group that manages the containers, full execution with 1.4.4: === # Apptainer version $ apptainer --version apptainer version 1.4.4-1.el9 # Container destination path ("outdir" variable) $ outdir="./my_dir/" # Dockerfile content $ cat fuse_err/Dockerfile FROM registry.access.redhat.com/ubi9/ubi:9.7 ENV LANG=C.UTF-8 \ LC_ALL=C.UTF-8 # Script to "fill" the image $ cat installer_examples/fuse_err_img.sh example text # As "user1" make container (.sif) $ echo ${USER} user1 $ podman build --cgroup-manager cgroupfs -t fuserr -f fuse_err/Dockerfile fuse_err/ STEP 1/2: FROM registry.access.redhat.com/ubi9/ubi:9.7 STEP 2/2: ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 --> Using cache 4919ea81dc0aaabb30657fe52dc3d1ec53f753aee5ff369880114b2904b7c576 COMMIT fuserr --> 4919ea81dc0a Successfully tagged localhost/fuserr:latest 4919ea81dc0aaabb30657fe52dc3d1ec53f753aee5ff369880114b2904b7c576 $ podman tag fuserr localhost:5000/fuserr $ podman save fuserr -o ${outdir}/fuserr.tar Copying blob 5a45357d82b6 done | Copying config 4919ea81dc done | Writing manifest to image destination $ apptainer build ${outdir}/fuserr.sif docker-archive:${outdir}/fuserr.tar INFO: Starting build... INFO: Fetching OCI image... INFO: Extracting OCI image... 2025/12/03 13:57:01 warn rootless{usr/bin/newgidmap} ignoring (usually) harmless EPERM on setxattr "security.capability" 2025/12/03 13:57:01 warn rootless{usr/bin/newuidmap} ignoring (usually) harmless EPERM on setxattr "security.capability" 2025/12/03 13:57:03 warn xattr{usr/bin/write} ignoring ENOTSUP on setxattr "user.rootlesscontainers" 2025/12/03 13:57:03 warn xattr{${outdir}/apptainer_tmpdir/build-temp-4238526265/rootfs/usr/bin/write} destination filesystem does not support xattrs, further warnings will be suppressed INFO: Inserting Apptainer configuration... INFO: Creating SIF file... [==============================================================================================================================================================] 100 % 0s INFO: Build complete: ${outdir}/fuserr.sif # As "user1" makes image (.img) $ echo ${USER} user1 $ apptainer overlay create --create-dir /opt/ --size 5000 fuse_err_overlay.img INFO: Creating overlay image for use without fakeroot. INFO: Consider re-running with --fakeroot option. $ chmod a+rx /clusterstorage/t-rcds/cpetrini/containers/fuserr.sif $ apptainer exec --bind ${PWD} --overlay fuse_err_overlay.img ${outdir}/fuserr.sif sh installer_examples/fuse_err_img.sh e2fsck -y -f fuse_err_overlay.img e2fsck 1.46.5 (30-Dec-2021) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information fuse_err_overlay.img: 15/320000 files (0.0% non-contiguous), 39003/1280000 blocks $ resize2fs -M fuse_err_overlay.img resize2fs 1.46.5 (30-Dec-2021) Resizing the filesystem on fuse_err_overlay.img to 17227 (4k) blocks. The filesystem on fuse_err_overlay.img is now 17227 (4k) blocks long. $ chmod a+rx fuse_err_overlay.img # As user1 run container loading image $ echo ${USERNAME} user1 $ apptainer shell --overlay fuse_err_overlay.img:ro ${outdir}/fuserr.sif Apptainer> ls /opt/ fuserr.txt Apptainer> # As "user2" run container loading the image (both owned by "user1", rx for "user2") $ echo ${USER} user2 $ apptainer shell --overlay fuse_err_overlay.img:ro ${outdir}/fuserr.sif FATAL: container creation failed: image driver mount failure: image driver fuse2fs instance exited with error: fuse2fs exited: FUSE2FS (3): Permission denied. FUSE2FS (3): Please run e2fsck -fy. ===
Thank you for those details. It turns out this is not an upstream problem, it is only a problem with the EPEL/Fedora build. The patch files that were supposed to fix this problem in 1.4.4 were empty in src.fedoraproject.org! I built 1.4.5 yesterday so will fix this in a 1.4.5-2.
FEDORA-2025-df330356b2 (apptainer-1.4.5-2.fc41) has been submitted as an update to Fedora 41. https://bodhi.fedoraproject.org/updates/FEDORA-2025-df330356b2
FEDORA-EPEL-2025-32198453b0 (apptainer-1.4.5-2.el10_2) has been submitted as an update to Fedora EPEL 10.2. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2025-32198453b0
FEDORA-EPEL-2025-473cf23bc7 (apptainer-1.4.5-2.el9) has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2025-473cf23bc7
FEDORA-EPEL-2025-8e02728fbe (apptainer-1.4.5-2.el10_1) has been submitted as an update to Fedora EPEL 10.1. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2025-8e02728fbe
FEDORA-2025-ff963b3775 (apptainer-1.4.5-2.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-ff963b3775
FEDORA-EPEL-2025-32198453b0 has been pushed to the Fedora EPEL 10.2 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2025-32198453b0 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2025-473cf23bc7 has been pushed to the Fedora EPEL 9 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2025-473cf23bc7 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2025-8e02728fbe has been pushed to the Fedora EPEL 10.1 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2025-8e02728fbe See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2025-44da2a402e has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2025-44da2a402e See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-cf169a01e8 has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-cf169a01e8` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-cf169a01e8 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-ff963b3775 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-ff963b3775` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-ff963b3775 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-df330356b2 has been pushed to the Fedora 41 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-df330356b2` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-df330356b2 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2025-32198453b0 (apptainer-1.4.5-2.el10_2) has been pushed to the Fedora EPEL 10.2 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2025-473cf23bc7 (apptainer-1.4.5-2.el9) has been pushed to the Fedora EPEL 9 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-ff963b3775 (apptainer-1.4.5-2.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2025-8e02728fbe (apptainer-1.4.5-2.el10_1) has been pushed to the Fedora EPEL 10.1 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-cf169a01e8 (apptainer-1.4.5-2.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2025-44da2a402e (apptainer-1.4.5-2.el8) has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-df330356b2 (apptainer-1.4.5-2.fc41) has been pushed to the Fedora 41 stable repository. If problem still persists, please make note of it in this bug report.
*** Bug 2423028 has been marked as a duplicate of this bug. ***
*** Bug 2423029 has been marked as a duplicate of this bug. ***
*** Bug 2423030 has been marked as a duplicate of this bug. ***
*** Bug 2423031 has been marked as a duplicate of this bug. ***
*** Bug 2423032 has been marked as a duplicate of this bug. ***