Description of problem: I receive an error any time I try to pull an image using podman 3.1.0-0.2.rc2 Error processing tar file(exit status 1): operation not permitted Error: Error committing the finished image: error adding layer with blob "sha256:cca21acb641a96561e0cf9a0c1c7b7ffbaaefc92185bd8a9440f6049c838e33b": Error processing tar file(exit status 1): operation not permitted Version-Release number of selected component (if applicable): podman-3.1.0-0.2.rc2.fc34.x86_64 How reproducible: Seems 100%, every time, ever image Steps to Reproduce: 1. Update to the latest podman on Fedora 34 2. Try to pull an image as a user Actual results: Error processing tar file(exit status 1): operation not permitted Error: Error committing the finished image: error adding layer with blob "sha256:cca21acb641a96561e0cf9a0c1c7b7ffbaaefc92185bd8a9440f6049c838e33b": Error processing tar file(exit status 1): operation not permitted Expected results: I can pull images successfully Additional info: I can still pull images as root Before upgrading: $ podman pull quay.io/konveyor/mig-operator-container:latest Trying to pull quay.io/konveyor/mig-operator-container:latest... Getting image source signatures Copying blob d9e72d058dc5 skipped: already exists Copying blob 20a01840033c done Copying blob 88c2d6a36b03 done Copying blob 79cbf6512af6 done Copying blob cca21acb641a done Copying blob 5ed6a26e4d1b done Copying blob 32e5fb4144d9 done Copying blob 6b326f51bce5 done Copying blob 481ff4ed57f1 done Copying blob 98595880ca71 done Copying blob 8107a6de225a done Copying blob b4d9866d7642 done Copying blob 6c184bf70b56 done Copying blob b276df7f5fe0 done Copying blob 80e180208ff6 done Copying blob a55e3d1667d6 done Copying blob 7d11dc19d6e9 done Copying blob 8373cd3f7e9a done Copying blob c27fcbad0f76 done Copying config a5824edea8 done Writing manifest to image destination Storing signatures a5824edea85b286524da7e634f67650842a51a60c5d284820396f206ca3b0df1 $ rpm -q podman podman-3.0.1-1.fc34.x86_64 $ sudo dnf -y update Last metadata expiration check: 0:17:20 ago on Thu 25 Mar 2021 03:08:08 PM EDT. Dependencies resolved. =================================================================================================================================================================================================================== Package Architecture Version Repository Size =================================================================================================================================================================================================================== Upgrading: podman x86_64 2:3.1.0-0.2.rc2.fc34 updates-testing 12 M podman-docker noarch 2:3.1.0-0.2.rc2.fc34 updates-testing 182 k podman-plugins x86_64 2:3.1.0-0.2.rc2.fc34 updates-testing 1.3 M Transaction Summary =================================================================================================================================================================================================================== Upgrade 3 Packages Total download size: 13 M Downloading Packages: (1/3): podman-docker-3.1.0-0.2.rc2.fc34.noarch.rpm 681 kB/s | 182 kB 00:00 (2/3): podman-plugins-3.1.0-0.2.rc2.fc34.x86_64.rpm 3.5 MB/s | 1.3 MB 00:00 (3/3): podman-3.1.0-0.2.rc2.fc34.x86_64.rpm 17 MB/s | 12 MB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 15 MB/s | 13 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Upgrading : podman-plugins-2:3.1.0-0.2.rc2.fc34.x86_64 1/6 Upgrading : podman-2:3.1.0-0.2.rc2.fc34.x86_64 2/6 Upgrading : podman-docker-2:3.1.0-0.2.rc2.fc34.noarch 3/6 Cleanup : podman-docker-2:3.0.1-1.fc34.noarch 4/6 Cleanup : podman-2:3.0.1-1.fc34.x86_64 5/6 Cleanup : podman-plugins-2:3.0.1-1.fc34.x86_64 6/6 Running scriptlet: podman-plugins-2:3.0.1-1.fc34.x86_64 6/6 Verifying : podman-2:3.1.0-0.2.rc2.fc34.x86_64 1/6 Verifying : podman-2:3.0.1-1.fc34.x86_64 2/6 Verifying : podman-docker-2:3.1.0-0.2.rc2.fc34.noarch 3/6 Verifying : podman-docker-2:3.0.1-1.fc34.noarch 4/6 Verifying : podman-plugins-2:3.1.0-0.2.rc2.fc34.x86_64 5/6 Verifying : podman-plugins-2:3.0.1-1.fc34.x86_64 6/6 Upgraded: podman-2:3.1.0-0.2.rc2.fc34.x86_64 podman-docker-2:3.1.0-0.2.rc2.fc34.noarch podman-plugins-2:3.1.0-0.2.rc2.fc34.x86_64 Complete After upgrading: $ podman pull quay.io/konveyor/mig-operator-container:latest Trying to pull quay.io/konveyor/mig-operator-container:latest... Getting image source signatures Copying blob 32e5fb4144d9 done Copying blob cca21acb641a done Copying blob 20a01840033c done Copying blob d9e72d058dc5 done Copying blob 88c2d6a36b03 done Copying blob 5ed6a26e4d1b done Copying blob 79cbf6512af6 done Copying blob 481ff4ed57f1 done Copying blob 6b326f51bce5 done Copying blob 98595880ca71 done Copying blob 8107a6de225a done Copying blob b4d9866d7642 done Copying blob 6c184bf70b56 done Copying blob b276df7f5fe0 done Copying blob 80e180208ff6 done Copying blob a55e3d1667d6 done Copying blob 7d11dc19d6e9 done Copying blob 8373cd3f7e9a done Copying blob c27fcbad0f76 done Copying config a5824edea8 done Writing manifest to image destination Storing signatures Error processing tar file(exit status 1): operation not permitted Error: Error committing the finished image: error adding layer with blob "sha256:cca21acb641a96561e0cf9a0c1c7b7ffbaaefc92185bd8a9440f6049c838e33b": Error processing tar file(exit status 1): operation not permitted $ rpm -q podman podman-3.1.0-0.2.rc2.fc34.x86_64 Downgrading allows it to work again
I compared podman info output between versions. After upgrading it looks like the graphOptions are gone @@ -76,15 +76,7 @@ running: 0 stopped: 0 graphDriverName: overlay - graphOptions: - overlay.mount_program: - Executable: /usr/bin/fuse-overlayfs - Package: fuse-overlayfs-1.4.0-3.fc34.x86_64 - Version: |- - fusermount3 version: 3.10.2 - fuse-overlayfs: version 1.4 - FUSE library version 3.10.2 - using FUSE kernel interface version 7.31 + graphOptions: {} graphRoot: /home/jason/.local/share/containers/storage graphStatus: Backing Filesystem: xfs I created $HOME/.config/containers/storage.conf and set the mount_program option and now I see the graphOptions restored and it works. [storage] driver = "overlay" rootless_storage_path = "$HOME/.local/share/containers/storage" [storage.options] additionalimagestores = [ ] [storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs" mountopt = "nodev,metacopy=on" [storage.options.thinpool]
Likely related to https://github.com/containers/podman/issues/9936 upstream. You are the first people we've had who can reproduce on Fedora. Can you verify that, if you remove storage.conf again, things go back to being broken?
With podman-3.1.0-2.fc34.x86_64 I no longer see the issue without the storage.conf I commented on this issue as I thought it might be the same https://github.com/containers/podman/issues/9834 I think it could have been: https://github.com/containers/storage/pull/841 It looked like from that issue the plan was to update containers/storage and if that happened between the -0.2rc2 and -2 package it very likely fixed it.