Description of problem: I had btrfs as a storage driver configured on Fedora 35 in ~/.config/containers/storage.conf: [storage] driver="btrfs" After upgrading to Fedora 36 (and finally convincing container-selinux to install successfully, see https://bugzilla.redhat.com/show_bug.cgi?id=2056303) I couldn't get my containers to run. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.Upgrade to Fedora 36 2. Add btrfs as storage driver in containers.conf if not already set 3. podman system reset 4. podman run fedora:35 bash Actual results: Error: error creating container storage: error creating read-write layer with ID "db743395667be6b494116c7ce7e11c4afda29cc875e3567a7040d0804ab1c6b1": setxattr /home/edwin/.local/share/containers/storage/btrfs/subvolumes/db743395667be6b494116c7ce7e11c4afda29cc875e3567a7040d0804ab1c6b1/etc/alternatives/libnssckbi.so.x86_64: operation not permitted Expected results: No errors, container working Additional info: I tried reinstalling container-selinux and podman multiple times (until they finally succeeded, see workaround on above bug). Also ran 'fixfiles -B onboot' and rebooted, but that didn't help solving this error, and neither did the 'podman system reset'. Workaround is to go back to the default driver instead of the btrfs one.
What AVC's are you seeing, and why are you using btrfs back end versus Overlay. Very few people use BTRFS backend and it is not well tested. Most people use overlay and it works well on BTRFS storage.
Good point about AVC: I don't see any. Interestingly even after 'sudo setenforce 0' I still get the same setxattr error. It doesn't fail on all containers, e.g. this one works: podman run busybox:latest sh This fails: podman run fedora:35 sh Error: error creating container storage: error creating read-write layer with ID "618dde56c3e89ccf7f2adbead049f75a070a7d2ecf0ed69f28a85599e31c649f": setxattr /home/edwin/.local/share/containers/storage/btrfs/subvolumes/618dde56c3e89ccf7f2adbead049f75a070a7d2ecf0ed69f28a85599e31c649f/etc/alternatives/libnssckbi.so.x86_64: operation not permitted strace shows this, so definetely SELinux related: [pid 51478] setxattr("/home/edwin/.local/share/containers/storage/btrfs/subvolumes/f9dbab13b9dacf4f005245e5342b60844d49ef59c7274b68d2d32bd2fd68f91 7/etc/alternatives/libnssckbi.so.x86_64", "security.selinux", "system_u:object_r:container_file_t:s0:c286,c1020", 48, 0 <unfinished ...> However doing that manually on a file works: $ touch x $ chcon system_u:object_r:container_file_t:s0:c286,c1020 x $ ls -lZ x -rwxr-xr-x. 1 edwin edwin system_u:object_r:container_file_t:s0:c286,c1020 8381752 Apr 1 22:28 x I think I see what the problem is though: $ ls -Zl /home/edwin/.local/share/containers/storage/btrfs/subvolumes/f9dbab13b9dacf4f005245e5342b60844d49ef59c7274b68d2d32bd2fd68f917/etc/alternatives/libnssckbi.so.x86_64 lrwxrwxrwx. 1 edwin edwin unconfined_u:object_r:unlabeled_t:s0 34 Feb 21 06:47 /home/edwin/.local/share/containers/storage/btrfs/subvolumes/f9dbab13b9dacf4f005245e5342b60844d49ef59c7274b68d2d32bd2fd68f917/etc/alternatives/libnssckbi.so.x86_64 -> /usr/lib64/pkcs11/p11-kit-trust.so $ chcon system_u:object_r:container_file_t:s0:c286,c1020 /home/edwin/.local/share/containers/storage/btrfs/subvolumes/f9dbab13b9dacf4f005245e5342b60844d49ef59c7274b68d2d32bd2fd68f917/etc/alternatives/libnssckbi.so.x86_64 chcon: failed to change context of '/home/edwin/.local/share/containers/storage/btrfs/subvolumes/f9dbab13b9dacf4f005245e5342b60844d49ef59c7274b68d2d32bd2fd68f917/etc/alternatives/libnssckbi.so.x86_64' to ‘system_u:object_r:container_file_t:s0:c286,c1020’: Operation not permitted That file is a symlink (to a file that happens to exist on the host system), and it looks like it is trying to set the context on the target of the symbolic link, and not the symbolic link itself (confusion between lsetxattr and setxattr)? Seems like a bug in the btrfs driver of podman that didn't exist in Fedora 35. Background on why I'm using btrfs storage driver: I've switched to the btrfs storage driver for podman to avoid fuse-overlayfs. However that isn't strictly necessary anymore because overlayfs has seen some improvements and should work rootless now without the need of fuse-overlayfs (https://www.redhat.com/sysadmin/podman-rootless-overlay). I haven't measured but I'd assume that using btrfs driver would avoid CoW, whereas my /home has CoW on by default (I can chattr the podman storage directory though to turn that off). Also I use snapper so unless I turn off CoW or configure something differently all the containers would be part of snapshots, using up space, whereas they can always be rebuilt/redownloaded and don't really need to be snapshotted.
FWIW this works, which confirms the lsetxattr vs setxattr suspicion I had: chcon --no-dereference system_u:object_r:container_file_t:s0:c286,c1020 /home/edwin/.local/share/containers/storage/btrfs/subvolumes/f9dbab13b9dacf4f005245e5342b60844d49ef59c7274b68d2d32bd2fd68f917/etc/alternatives/libnssckbi.so.x86_64
Could you build a podman with this change and see if BTRFS works for you?
Actually a bug was just fixed in opencontainers/selinux that fixes the problem you are seeing. https://github.com/opencontainers/selinux/pull/173
Thanks for locating the patch, I've done a test build locally using 'fedpkg mockbuild' (https://github.com/edwintorok/podman.spec-tmp/commit/8ec93319b940c2f53098cd9db33e69a12d792fca is my attempt at backporting, had to replace paths in the patch to modify the vendored files). This works now: ``` podman run fedora:35 sh -c 'echo hello' hello ```