Recent versions of Silverblue and Kinoite have permissions issues running podman, see e.g. roth@dell-c2-bf-b4:/var/home/roth$ podman images ERRO[0000] running `/usr/bin/newuidmap 22530 0 1017 1 1 524288 65536 65537 589824 65536`: newuidmap: write to uid_map failed: Operation not permitted Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1 I'm not sure what package this regression is actually in but it's initially noticed in podman. Reproducible: Always Steps to Reproduce: 1. update to at least Rawhide.20231122.n.0 2. run podman as non-root to e.g. list containers or images 3. Actual Results: Podman reports permission errors Expected Results: Podman should work as expected I found this on both Silverblue and Kinoite. I bisected the release tags for Kinoite and found that the last working version is Rawhide.20231120.n.0. I'm listing this as high severity since immutable distributions like Kinoite/Silverblue/CoreOS are not very functional without podman (and toolbox). See also: [non working image] roth@dell-c2-bf-b4:/var/home/roth$ rpm -q podman podman-4.7.2-1.fc40.x86_64 roth@dell-c2-bf-b4:/var/home/roth$ rpm -qf /usr/bin/newuidmap shadow-utils-4.14.0-2.fc40.x86_64 roth@dell-c2-bf-b4:/var/home/roth$ rpm-ostree status State: idle AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 4h 2min ago Deployments: fedora:fedora/rawhide/x86_64/kinoite Version: Rawhide.20231126.n.0 (2023-11-26T06:08:02Z) BaseCommit: 999f3f66a5e665711255ea7d1a79bc8bab03015dfd219ed0dd21af2de16f597d GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC Diff: 69 upgraded, 1 removed, 2 added LayeredPackages: efitools gstreamer1-plugin-openh264 gstreamer1-plugins-ugly kwalletcli pass pesign sbsigntools LocalPackages: akmods-keys-0.0.2-8.fc40.noarch rpmfusion-free-release-40-0.1.noarch rpmfusion-nonfree-release-40-0.1.noarch Initramfs: regenerate ● fedora:fedora/rawhide/x86_64/kinoite Version: Rawhide.20231122.n.0 (2023-11-22T06:05:21Z) BaseCommit: 5d60dec96a6a645c987974776566ac33a6f2f0a18197e361d5ae12ca09bf2409 GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC LayeredPackages: efitools gstreamer1-plugin-openh264 gstreamer1-plugins-ugly kwalletcli pass pesign sbsigntools LocalPackages: akmods-keys-0.0.2-8.fc40.noarch rpmfusion-free-release-40-0.1.noarch rpmfusion-nonfree-release-40-0.1.noarch Initramfs: regenerate fedora:fedora/rawhide/x86_64/kinoite Version: Rawhide.20231120.n.0 (2023-11-20T06:08:27Z) BaseCommit: 4f41da6118c373bdcc9237cc7109c395ad23e9465eb430f60c8aee1f64ca11bd GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC LayeredPackages: efitools gstreamer1-plugin-openh264 gstreamer1-plugins-ugly kwalletcli pass pesign sbsigntools LocalPackages: akmods-keys-0.0.2-8.fc40.noarch rpmfusion-free-release-40-0.1.noarch rpmfusion-nonfree-release-40-0.1.noarch Initramfs: regenerate Pinned: yes roth@dell-c2-bf-b4:/var/home/roth$ rpm-ostree db diff ostree diff commit from: booted deployment (498577e0b98bfce424226c481550ce52366340023c761d2e209164c6bd0acd0a) ostree diff commit to: pending deployment (f9f63221a8d4e1d404e330a9f708587560343adc2aa0700949e3fb5322d3a43c) Upgraded: btrfs-progs 6.5.1-1.fc40 -> 6.6.2-1.fc40 checkpolicy 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 chrony 4.4-1.fc40 -> 4.5-0.1.pre1.fc40 crun 1.11.2-1.fc40 -> 1.12-1.fc40 dmraid 1.0.0.rc16-56.fc39 -> 1.0.0.rc16-57.fc40 dmraid-events 1.0.0.rc16-56.fc39 -> 1.0.0.rc16-57.fc40 dmraid-libs 1.0.0.rc16-56.fc39 -> 1.0.0.rc16-57.fc40 git-core 2.42.1-1.fc40 -> 2.43.0-1.fc40 git-core-doc 2.42.1-1.fc40 -> 2.43.0-1.fc40 ibus 1.5.29~rc2-2.fc40 -> 1.5.29~rc2-3.fc40 ibus-gtk2 1.5.29~rc2-2.fc40 -> 1.5.29~rc2-3.fc40 ibus-gtk3 1.5.29~rc2-2.fc40 -> 1.5.29~rc2-3.fc40 ibus-gtk4 1.5.29~rc2-2.fc40 -> 1.5.29~rc2-3.fc40 ibus-libs 1.5.29~rc2-2.fc40 -> 1.5.29~rc2-3.fc40 ibus-panel 1.5.29~rc2-2.fc40 -> 1.5.29~rc2-3.fc40 ibus-setup 1.5.29~rc2-2.fc40 -> 1.5.29~rc2-3.fc40 ipp-usb 0.9.23-3.fc39 -> 0.9.23-5.fc40 jbigkit-libs 2.1-26.fc39 -> 2.1-27.fc40 kde-connect 23.08.2-1.fc40 -> 23.08.2-2.fc40 kde-connect-libs 23.08.2-1.fc40 -> 23.08.2-2.fc40 kdeconnectd 23.08.2-1.fc40 -> 23.08.2-2.fc40 kernel 6.7.0-0.rc2.22.fc40 -> 6.7.0-0.rc2.20231125git0f5cc96c367f.26.fc40 kernel-core 6.7.0-0.rc2.22.fc40 -> 6.7.0-0.rc2.20231125git0f5cc96c367f.26.fc40 kernel-modules 6.7.0-0.rc2.22.fc40 -> 6.7.0-0.rc2.20231125git0f5cc96c367f.26.fc40 kernel-modules-core 6.7.0-0.rc2.22.fc40 -> 6.7.0-0.rc2.20231125git0f5cc96c367f.26.fc40 kernel-modules-extra 6.7.0-0.rc2.22.fc40 -> 6.7.0-0.rc2.20231125git0f5cc96c367f.26.fc40 kf6-filesystem 5.245.0-1.fc40 -> 5.245.0-2.fc40 libdbusmenu 16.04.0-23.fc40 -> 16.04.0-25.fc40 libdbusmenu-gtk3 16.04.0-23.fc40 -> 16.04.0-25.fc40 libibverbs 48.0-1.fc40 -> 48.0-2.fc40 libselinux 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 libselinux-utils 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 libsemanage 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 libsepol 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 liburing 2.4-3.fc39 -> 2.5-1.fc40 libvncserver 0.9.13-15.fc39 -> 0.9.14-1.fc40 libxml2 2.12.0-1.fc40 -> 2.12.1-1.fc40 mesa-dri-drivers 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 mesa-filesystem 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 mesa-libEGL 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 mesa-libGL 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 mesa-libgbm 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 mesa-libglapi 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 mesa-va-drivers 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 mesa-vulkan-drivers 23.3.0~rc2-3.fc40 -> 23.3.0~rc2-6.fc40 perl-Math-BigInt 1:2.0010.00-1.fc40 -> 1:2.0010.01-1.fc40 perl-Term-ANSIColor 5.01-501.fc39 -> 5.01-502.fc40 policycoreutils 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 policycoreutils-python-utils 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 python3-libselinux 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 python3-libsemanage 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 python3-policycoreutils 3.6-0.rc1.1.fc40 -> 3.6-0.rc2.1.fc40 sqlite 3.44.0-1.fc40 -> 3.44.1-1.fc40 sqlite-libs 3.44.0-1.fc40 -> 3.44.1-1.fc40 system-config-printer-libs 1.5.18-5.fc39 -> 1.5.18-6.fc40 system-config-printer-udev 1.5.18-5.fc39 -> 1.5.18-6.fc40 systemd 255~rc2-1.fc40 -> 255~rc3-1.fc40 systemd-libs 255~rc2-1.fc40 -> 255~rc3-1.fc40 systemd-networkd 255~rc2-1.fc40 -> 255~rc3-1.fc40 systemd-oomd-defaults 255~rc2-1.fc40 -> 255~rc3-1.fc40 systemd-pam 255~rc2-1.fc40 -> 255~rc3-1.fc40 systemd-resolved 255~rc2-1.fc40 -> 255~rc3-1.fc40 systemd-udev 255~rc2-1.fc40 -> 255~rc3-1.fc40 vim-data 2:9.0.2105-1.fc40 -> 2:9.0.2120-1.fc40 vim-minimal 2:9.0.2105-1.fc40 -> 2:9.0.2120-1.fc40 wireplumber 0.4.15-2.fc40 -> 0.4.16-1.fc40 wireplumber-libs 0.4.15-2.fc40 -> 0.4.16-1.fc40 xdg-desktop-portal 1.18.1-2.fc40 -> 1.18.2-1.fc40 xdg-utils 1.1.3-15.fc40 -> 1.2.0~git20231511.21fb316-1.fc40 Removed: pulseaudio-qt-1.3-5.fc39.x86_64 Added: amd-ucode-firmware-20231111-1.fc40.noarch pulseaudio-qt-qt5-1.3^20231120.081305.36f5625-2.fc40.x86_64 [non-working image] roth@dell-c2-bf-b4:/var/home/roth$ ls -alZ /usr/bin/newuidmap -rwxr-xr-x. 3 root root system_u:object_r:bin_t:s0 43296 Dec 31 1969 /usr/bin/newuidmap roth@dell-c2-bf-b4:/var/home/roth$ getfattr /usr/bin/newuidmap roth@dell-c2-bf-b4:/var/home/roth$ getfacl /usr/bin/newuidmap getfacl: Removing leading '/' from absolute path names # file: usr/bin/newuidmap # owner: root # group: root user::rwx group::r-x other::r-x
I think my 'db diff' output is incorrect. I un-deployed the unused deployments and this may be more useful: roth@dell-c2-bf-b4:/var/home/roth$ rpm-ostree db diff ostree diff commit from: rollback deployment (77a9daf50e9f4e4a3a1f0c3f5a82bb137e92bdc9de204196be1d5aee6ffd74ee) ostree diff commit to: booted deployment (498577e0b98bfce424226c481550ce52366340023c761d2e209164c6bd0acd0a) Upgraded: bash 5.2.21-1.fc40 -> 5.2.21-2.fc40 bind-libs 32:9.18.19-1.fc40 -> 32:9.18.20-1.fc40 bind-license 32:9.18.19-1.fc40 -> 32:9.18.20-1.fc40 bind-utils 32:9.18.19-1.fc40 -> 32:9.18.20-1.fc40 bluedevil 5.27.9-1.fc40 -> 5.27.9-2.fc40 c-ares 1.21.0-1.fc40 -> 1.22.1-1.fc40 firefox 120.0-1.fc40 -> 120.0-2.fc40 firefox-langpacks 120.0-1.fc40 -> 120.0-2.fc40 fwupd 1.9.8-1.fc40 -> 1.9.9-1.fc40 fwupd-plugin-flashrom 1.9.8-1.fc40 -> 1.9.9-1.fc40 fwupd-plugin-modem-manager 1.9.8-1.fc40 -> 1.9.9-1.fc40 fwupd-plugin-uefi-capsule-data 1.9.8-1.fc40 -> 1.9.9-1.fc40 gnome-keyring 42.1-5.fc40 -> 42.1-7.fc40 gnome-keyring-pam 42.1-5.fc40 -> 42.1-7.fc40 gstreamer1 1.22.7-1.fc40 -> 1.22.7-2.fc40 gstreamer1-plugins-bad-free 1.22.7-1.fc40 -> 1.22.7-2.fc40 gstreamer1-plugins-bad-free-libs 1.22.7-1.fc40 -> 1.22.7-2.fc40 kernel 6.7.0-0.rc1.20231117git7475e51b8796.19.fc40 -> 6.7.0-0.rc2.22.fc40 kernel-core 6.7.0-0.rc1.20231117git7475e51b8796.19.fc40 -> 6.7.0-0.rc2.22.fc40 kernel-modules 6.7.0-0.rc1.20231117git7475e51b8796.19.fc40 -> 6.7.0-0.rc2.22.fc40 kernel-modules-core 6.7.0-0.rc1.20231117git7475e51b8796.19.fc40 -> 6.7.0-0.rc2.22.fc40 kernel-modules-extra 6.7.0-0.rc1.20231117git7475e51b8796.19.fc40 -> 6.7.0-0.rc2.22.fc40 kf5-kglobalaccel 5.111.0-6.fc40 -> 5.111.0-8.fc40 kf5-kglobalaccel-libs 5.111.0-6.fc40 -> 5.111.0-8.fc40 kf6-baloo-file 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-baloo-libs 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-kio-core 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-kio-core-libs 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-kio-doc 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-kio-file-widgets 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-kio-gui 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-kio-widgets 5.245.0-1.fc40 -> 5.245.0-2.fc40 kf6-kio-widgets-libs 5.245.0-1.fc40 -> 5.245.0-2.fc40 kpartx 0.9.6-1.fc40 -> 0.9.7-1.fc40 libcap 2.48-7.fc39 -> 2.69-1.fc40 libwnck3 43.0-5.fc39 -> 43.0-6.fc40 openvpn 2.6.7-1.fc40 -> 2.6.8-1.fc40 pam 1.5.3-7.fc40 -> 1.5.3-8.fc40 pam-libs 1.5.3-7.fc40 -> 1.5.3-8.fc40 perl-MIME-Base64 3.16-500.fc39 -> 3.16-501.fc40 plymouth 22.02.122-5.fc39 -> 22.02.122-6.fc40 plymouth-core-libs 22.02.122-5.fc39 -> 22.02.122-6.fc40 plymouth-graphics-libs 22.02.122-5.fc39 -> 22.02.122-6.fc40 plymouth-plugin-label 22.02.122-5.fc39 -> 22.02.122-6.fc40 plymouth-plugin-two-step 22.02.122-5.fc39 -> 22.02.122-6.fc40 plymouth-scripts 22.02.122-5.fc39 -> 22.02.122-6.fc40 plymouth-system-theme 22.02.122-5.fc39 -> 22.02.122-6.fc40 plymouth-theme-spinner 22.02.122-5.fc39 -> 22.02.122-6.fc40 polkit 123-1.fc39 -> 123-3.fc40 polkit-libs 123-1.fc39 -> 123-3.fc40 systemd 254.5-2.fc40 -> 255~rc2-1.fc40 systemd-libs 254.5-2.fc40 -> 255~rc2-1.fc40 systemd-networkd 254.5-2.fc40 -> 255~rc2-1.fc40 systemd-oomd-defaults 254.5-2.fc40 -> 255~rc2-1.fc40 systemd-pam 254.5-2.fc40 -> 255~rc2-1.fc40 systemd-resolved 254.5-2.fc40 -> 255~rc2-1.fc40 systemd-udev 254.5-2.fc40 -> 255~rc2-1.fc40 Removed: binutils-2.41-13.fc40.x86_64 binutils-gold-2.41-13.fc40.x86_64 cmake-3.27.7-1.fc40.x86_64 cmake-data-3.27.7-1.fc40.noarch gc-8.2.2-4.fc39.x86_64 gcc-13.2.1-5.fc40.x86_64 gcc-c++-13.2.1-5.fc40.x86_64 glibc-devel-2.38.9000-22.fc40.x86_64 glibc-headers-x86-2.38.9000-22.fc40.noarch guile22-2.2.7-9.fc39.x86_64 jsoncpp-1.9.5-5.fc39.x86_64 kernel-headers-6.7.0-0.rc1.git0.1.fc40.x86_64 kf6-kauth-devel-5.245.0-1.fc40.x86_64 kf6-kbookmarks-devel-5.245.0-1.fc40.x86_64 kf6-kcodecs-devel-5.245.0-1.fc40.x86_64 kf6-kcolorscheme-devel-5.245.0-1.fc40.x86_64 kf6-kcompletion-devel-5.245.0-1.fc40.x86_64 kf6-kconfig-devel-5.245.0-1.fc40.x86_64 kf6-kconfigwidgets-devel-5.245.0-2.fc40.x86_64 kf6-kcoreaddons-devel-5.245.0-1.fc40.x86_64 kf6-kded-5.245.0-1.fc40.x86_64 kf6-kio-5.245.0-1.fc40.x86_64 kf6-kio-devel-5.245.0-1.fc40.x86_64 kf6-kitemviews-devel-5.245.0-1.fc40.x86_64 kf6-kjobwidgets-devel-5.245.0-1.fc40.x86_64 kf6-kservice-devel-5.245.0-1.fc40.x86_64 kf6-kwidgetsaddons-devel-5.245.0-1.fc40.x86_64 kf6-kwindowsystem-devel-5.245.0-1.fc40.x86_64 kf6-kxmlgui-devel-5.245.0-1.fc40.x86_64 kf6-solid-devel-5.245.0-1.fc40.x86_64 libX11-devel-1.8.7-1.fc40.x86_64 libXau-devel-1.0.11-4.fc40.x86_64 libXvMC-1.0.13-3.fc39.x86_64 libdb-convert-util-5.3.28-58.fc40.x86_64 libfbclient2-4.0.4.3010-1.fc40.x86_64 libglvnd-core-devel-1:1.7.0-1.fc40.x86_64 libglvnd-devel-1:1.7.0-1.fc40.x86_64 libpq-15.3-1.fc39.x86_64 libstdc++-devel-13.2.1-5.fc40.x86_64 libtommath-1.2.1-1.fc40.x86_64 libxcb-devel-1.16-2.fc40.x86_64 libxcrypt-devel-4.4.36-2.fc39.x86_64 libxkbcommon-devel-1.6.0-1.fc40.x86_64 libxml2-devel-2.12.0-1.fc40.x86_64 make-1:4.4.1-2.fc39.x86_64 mesa-libEGL-devel-23.3.0~rc2-3.fc40.x86_64 mesa-libxatracker-23.3.0~rc2-3.fc40.x86_64 qt6-qtbase-devel-6.6.0-7.fc40.x86_64 qt6-qtbase-ibase-6.6.0-7.fc40.x86_64 qt6-qtbase-mysql-6.6.0-7.fc40.x86_64 qt6-qtbase-odbc-6.6.0-7.fc40.x86_64 qt6-qtbase-postgresql-6.6.0-7.fc40.x86_64 qt6-rpm-macros-6.6.0-1.fc40.noarch rhash-1.4.3-3.fc39.x86_64 unixODBC-2.3.11-4.fc39.x86_64 vim-filesystem-2:9.0.2105-1.fc40.noarch vulkan-headers-1.3.261.1-2.fc40.noarch vulkan-loader-devel-1.3.261.1-1.fc40.x86_64 xorg-x11-drv-amdgpu-23.0.0-2.fc39.x86_64 xorg-x11-drv-ati-19.1.0-10.fc39.x86_64 xorg-x11-drv-evdev-2.10.6-14.fc39.x86_64 xorg-x11-drv-fbdev-0.5.0-13.fc39.x86_64 xorg-x11-drv-intel-2.99.917-56.20210115.fc39.x86_64 xorg-x11-drv-nouveau-1:1.0.17-6.fc39.x86_64 xorg-x11-drv-openchrome-0.6.400-6.20210215git5dbad06.fc39.x86_64 xorg-x11-drv-qxl-0.1.6-2.fc39.x86_64 xorg-x11-drv-vesa-2.5.0-6.fc39.x86_64 xorg-x11-drv-vmware-13.4.0-2.fc39.x86_64 xorg-x11-drv-wacom-1.2.0-2.fc39.x86_64 xorg-x11-drv-wacom-serial-support-1.2.0-2.fc39.x86_64 xorg-x11-proto-devel-2023.2-3.fc40.noarch xz-devel-5.4.5-1.fc40.x86_64 zlib-devel-1.2.13-5.fc40.x86_64 Added: bluez-obexd-5.70-3.fc40.x86_64
*** Bug 2252019 has been marked as a duplicate of this bug. ***
@dustymabe do you know the right place to redirect this bz? I can't reproduce this on regular Fedora, seems like an immutable only thing.
I just booted the latet build of Fedora CoreOS and don't seem to have any problems: ``` Fedora CoreOS 40.20231129.91.0 [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE [core@cosa-devsh ~]$ podman run -it --rm quay.io/fedora/fedora:latest Trying to pull quay.io/fedora/fedora:latest... Getting image source signatures Copying blob 718a00fe3212 done | Copying config 368a084ba1 done | Writing manifest to image destination [root@6da5b36edbd0 /]# id uid=0(root) gid=0(root) groups=0(root) [root@6da5b36edbd0 /]# cat /etc/redhat-release Fedora release 39 (Thirty Nine) [root@6da5b36edbd0 /]# [root@6da5b36edbd0 /]# exit exit [core@cosa-devsh ~]$ rpm -q podman podman-4.7.2-1.fc40.x86_64 ``` Maybe it's an issue specifically with silverblue or some sort of issue for upgrading systems? Can this be reproduced with a fresh install?
@debarshir PTAL as well for silverblue.
Just retried with kinoite Rawhide.20231130.n.0 and the problem persists
(In reply to Carl Roth from comment #6) > Just retried with kinoite Rawhide.20231130.n.0 and the problem persists does it work with selinux turned off (if it can be turned off at all on these envs) ?
I turned selinux off with 'sudo setenforce 0' and the problem persists
there is a minor polkit change between the working- and non-working versions that does some enablement of sysusers.d, not sure if that is related https://koji.fedoraproject.org/koji/buildinfo?buildID=2322558
(In reply to Lokesh Mandvekar from comment #5) > @debarshir PTAL as well for silverblue. I haven't been able to dig into it, but Toolbx's upstream CI is working across Fedora 37 to Rawhide with package-based Fedoras: https://github.com/containers/toolbox/pull/1376 https://github.com/containers/toolbox/pull/1412 I will take a closer look.
I also spotted this issue on my Silverblue system after updating after ~2 weeks (so a bit of a wide window but that bisect was already done above). ``` ● fedora:fedora/rawhide/x86_64/silverblue Version: Rawhide.20231123.n.0 (2023-11-23T06:08:29Z) BaseCommit: d8c2af43fff1e2eca0dae6bfcadd3fb4dcd1553d870e1a54d9571c3942e98a21 GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC RemovedBasePackages: firefox firefox-langpacks 120.0-2.fc40 LayeredPackages: bcc bpftool bpftrace ddcutil direnv fish flamegraph gdb gtkmm4.0 intel-gpu-tools libbpf-tools libpfm libva-utils ltrace meson pass perf powertop python3-system-calls scrcpy simple-scan strace sysprof tmux Pinned: yes fedora:fedora/rawhide/x86_64/silverblue Version: Rawhide.20231107.n.0 (2023-11-07T06:20:52Z) BaseCommit: cacc391f9b57670764403fa39b6ac666f51f512025615177da778e8f9d0ea880 GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC RemovedBasePackages: firefox firefox-langpacks 119.0-4.fc40 LayeredPackages: bcc bpftool bpftrace ddcutil direnv fish flamegraph gdb gtkmm4.0 intel-gpu-tools libbpf-tools libpfm libva-utils ltrace meson pass perf powertop python3-system-calls scrcpy simple-scan strace sysprof tmux Pinned: yes ``` I tried to reset SELinux labels as per Silverblue [instructions](https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_selinux_problems) and also tried to switch SELinux from enforcing to permissive. No change. I enabled debug logging in Polkit and no suspicious messages are in the logs. I saw some related issues about this upstream (e.g., https://github.com/containers/podman/issues/20336) where a question raised was what are the capabilities of the files: ``` ; getcap /usr/bin/newuidmap /usr/bin/newgidmap /usr/bin/newuidmap cap_setuid=ip /usr/bin/newgidmap cap_setgid=ip ``` while Dan Walsh in his example had this output ``` $ getcap /usr/bin/newuidmap /usr/bin/newgidmap /usr/bin/newuidmap cap_setuid=ep /usr/bin/newgidmap cap_setgid=ep ``` So, I tried running the following: ``` ; sudo setcap cap_setuid=ep /usr/bin/newuidmap ; sudo setcap cap_setuid=ep /usr/bin/newgidmap ``` but the error remains the same. I can not get an output of `podman info` without root due to the discussed problem but just in case the output of the command when run as root is: ``` host: arch: amd64 buildahVersion: 1.32.0 cgroupControllers: - cpuset - cpu - io - memory - hugetlb - pids - rdma - misc cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.8-2.fc40.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.8, commit: ' cpuUtilization: idlePercent: 73.17 systemPercent: 6.48 userPercent: 20.36 cpus: 4 databaseBackend: boltdb distribution: distribution: fedora variant: silverblue version: "40" eventLogger: journald freeLocks: 2038 hostname: marty-t460 idMappings: gidmap: null uidmap: null kernel: 6.7.0-0.rc2.20231122gitc2d5304e6c64.23.fc40.x86_64 linkmode: dynamic logDriver: journald memFree: 4785639424 memTotal: 16610889728 networkBackend: netavark networkBackendInfo: backend: netavark dns: package: aardvark-dns-1.8.0-1.fc40.x86_64 path: /usr/libexec/podman/aardvark-dns version: aardvark-dns 1.8.0 package: netavark-1.8.0-2.fc40.x86_64 path: /usr/libexec/podman/netavark version: netavark 1.8.0 ociRuntime: name: crun package: crun-1.11.2-1.fc40.x86_64 path: /usr/bin/crun version: |- crun version 1.11.2 commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d rundir: /run/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL os: linux pasta: executable: /usr/bin/pasta package: passt-0^20231119.g4f1709d-1.fc40.x86_64 version: | pasta 0^20231119.g4f1709d-1.fc40.x86_64 Copyright Red Hat GNU General Public License, version 2 or later <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket: exists: false path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.2-1.fc40.x86_64 version: |- slirp4netns version 1.2.2 commit: 0ee2d87523e906518d34a6b423271e4826f71faf libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.3 swapFree: 8589930496 swapTotal: 8589930496 uptime: 0h 16m 17.00s plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /usr/share/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev,metacopy=on graphRoot: /var/lib/containers/storage graphRootAllocated: 498403901440 graphRootUsed: 474421231616 graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "false" Supports d_type: "true" Supports shifting: "true" Supports volatile: "true" Using metacopy: "true" imageCopyTmpDir: /var/tmp imageStore: number: 4 runRoot: /run/containers/storage transientStore: false volumePath: /var/lib/containers/storage/volumes version: APIVersion: 4.7.2 Built: 1698762097 BuiltTime: Tue Oct 31 16:21:37 2023 GitCommit: "" GoVersion: go1.21.3 Os: linux OsArch: linux/amd64 Version: 4.7.2 ```
On a Silverblue 39 deployment the capabilities on the two binaries are different: ``` ; getcap /usr/bin/newuidmap /usr/bin/newgidmap /usr/bin/newuidmap cap_setuid=ep /usr/bin/newgidmap cap_setgid=ep ```
Another error with a database mismatch in addition to the error with newuidmap on today's rawhide (Rawhide.20231201.n.0). ``` /home/codebam podman ps -a ERRO[0000] running `/bin/newuidmap 3493 0 1000 1 1 524288 65536`: newuidmap: write to uid_map failed: Operation not permitted Error: cannot set up namespace using "/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1 /home/codebam podman system migrate ERRO[0000] Rolling back transaction to validate database: sql: transaction has already been committed or rolled back Error: database static dir "/home/codebam/.local/share/containers/storage/libpod" does not match our static dir "/var/home/codebam/.local/share/containers/storage/libpod": database configuration mismatch ```
That libpod error is being tracked at https://github.com/containers/podman/issues/20872
Problem still exists in the 2023-12-09 release but I am able patch it locally: oth@hp-14-dq0xxx:~$ rpm-ostree status State: idle AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot Deployments: ● fedora:fedora/rawhide/x86_64/silverblue Version: Rawhide.20231209.n.0 (2023-12-09T06:10:50Z) BaseCommit: e44f3b3f091788b968cfefab46bd37fe74fe82d67fd54e2a13edb41bacd1712d GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC LayeredPackages: gnome-tweaks gstreamer1-plugin-openh264 gstreamer1-plugins-ugly guestfs-tools libvirt-daemon pam_yubico virt-install virt-manager virt-top virt-viewer ykclient ykpers yubico-piv-tool yubikey-manager yubikey-personalization-gui LocalPackages: rpmfusion-free-release-40-0.1.noarch rpmfusion-nonfree-release-40-0.1.noarch Initramfs: --force-add tpm2-tss roth@hp-14-dq0xxx:~$ getcap /usr/bin/newuidmap /usr/bin/newuidmap cap_setuid=ip ^^ roth@hp-14-dq0xxx:~$ sudo ostree admin unlock Development mode enabled. A writable overlayfs is now mounted on /usr. All changes there will be discarded on reboot. The solution here appears to be clear -- who is responsible upstream for making these file capabilities changes? roth@hp-14-dq0xxx:~$ sudo setcap cap_setuid=ep /usr/bin/newuidmap roth@hp-14-dq0xxx:~$ sudo setcap cap_setgid=ep /usr/bin/newgidmap roth@hp-14-dq0xxx:~$ getcap /usr/bin/newuidmap /usr/bin/newuidmap cap_setuid=ep ^^ roth@hp-14-dq0xxx:~$ getcap /usr/bin/newgidmap /usr/bin/newgidmap cap_setgid=ep ^^ roth@hp-14-dq0xxx:~$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE localhost/ursus-ubuntu 22.04 061b427f0575 6 days ago 966 MB ... [podman command works now]
I have encountered an issue with one of my podman quadlet containers, which ceased functioning. Upon investigation, I discovered that the container was not generated until the "Group" line was removed. Could this issue be interrelated? " Group= The (numeric) GID to run as inside the container. This does not need to match the GID on the host, which can be modified with UsersNS, but if that is not specified, this GID is also used on the host. "
spec from shadow-utils at https://src.fedoraproject.org/rpms/shadow-utils/raw/rawhide/f/shadow-utils.spec appears to be correct ... %{_bindir}/lastlog %attr(4755,root,root) %{_bindir}/newgrp %attr(0755,root,root) %caps(cap_setgid=ep) %{_bindir}/newgidmap %attr(0755,root,root) %caps(cap_setuid=ep) %{_bindir}/newuidmap %{_sbindir}/adduser ...
It's not just newuidmap & newgidmap that have incorrect file capabilities. On my Silverblue (Rawhide.20231210.n.0) system, rpm knows these files have wrong capabilities: $ rpm -Va | grep 'P ' .......TP /usr/bin/newgidmap .......TP /usr/bin/newuidmap .......TP /usr/libexec/gstreamer-1.0/gst-ptp-helper .......TP /usr/bin/arping .......TP /usr/bin/clockdiff ..?...GTP /usr/sbin/suexec .......TP /usr/sbin/mtr-packet I guess the bug is in ostree or rpm-ostree.
This indeed looks like an rpm-ostree bug, likely introduced in 2023.12. I'm still investigating. It looks like this only impacts commits built with rpm-ostree 2023.12 and later, using `rpm-ostree compose tree` which would explain this is only impacting Silverblue/Kinoite/etc rawhide where this version of rpm-ostree has been pushed to the buildroot. Commits made via `rpm-ostree compose image` do not have this issue.
I can not reproduce the issue building Fedora CoreOS with 2024.01 so I'm not sure my analysis is correct.
Tracked upstream in https://github.com/coreos/rpm-ostree/issues/4765. Still investigating.
Fix by Jonathan in https://github.com/coreos/rpm-ostree/pull/4769. This will be fixed in Atomic Desktops and IoT once a new release of rpm-ostree is made, the package is updated in Fedora and the compose hosts are update to use this new version.
I've verified that this is fixed in Silverblue via rpm-ostree 2024.2 (https://bodhi.fedoraproject.org/updates/FEDORA-2024-78044caba3) which landed in Rawhide. Thanks for the report and investigation.