Overnight, the openQA install_default_update_ostree test started failing on all updates. It turns out it's failing because `toolbox -y enter` doesn't work: [root@fedora update_repo]# toolbox -y enter Error: failed to get the Podman version [root@fedora update_repo]# the failure looks a little odd in openQA because we aren't actually able to check the return code of the toolbox command, so we actually fail a bit later when we try to run `dnf` (assuming we're in the toolbox container) and it fails, but that's what's going on. This first failed on the containers-common-0.57.1-5.fc40 update, but since this isn't a gating test, that update went stable and this is now affecting all updates. I'm not 100% sure containers-common-0.57.1-5.fc40 actually caused it, but it seems possible, so filing against that at least for now, but CCing toolbox maintainer.
Is there any way you could run: # toolbox --verbose -y enter If so, what does this say: # podman version --format json
I'm unable to reproduce this on a fresh rawhide vm. $ rpm -q podman toolbox containers-common podman-4.9.0-1.fc40.x86_64 toolbox-0.0.99.5-3.fc40.x86_64 containers-common-0.57.1-1.fc40.noarch $ toolbox -y enter Welcome to the Toolbox; a container where you can install and run all your tools. - Use DNF in the usual manner to install command line tools. - To create a new tools container, run 'toolbox create'. For more information, see the documentation. ⬢[lsm5@toolbox ~]$
ahh nvm, gotta check with c/common -5 build. I'll reply back.
Bumped to containers-common 0.57.1-5. `toolbox -y enter` still works on rawhide for me.
It's still failing on every openQA test run. Note openQA is testing a fresh install of a freshly-built Silverblue installer image. This is likely not what you're testing, and may be significant. I tested locally, and you should be able to reproduce with https://adamwill.fedorapeople.org/02381092-FEDORA-2024-063990bca2-Silverblue-ostree-x86_64.iso (do a fresh install of that and then reproduce from the installed system). Additionally, it turns out that it fails as root, but not as a regular user: test@fedora:~$ toolbox -y enter Pulling registry.fedoraproject.org/fedora-toolbox:40: \^C test@fedora:~$ sudo su root@fedora:/var/home/test# toolbox -y enter Error: failed to get the Podman version
So, looking at the toolbox source, I see it runs `podman version --format json` to get the Podman version. And indeed, that works as regular user but fails as root too: test@fedora:~$ podman version --format json {"Client":{"APIVersion":"4.9.0","Version":"4.9.0","GoVersion":"go1.21.6","GitCommit":"","BuiltTime":"Wed Jan 24 02:03:24 2024","Built":1706090604,"OsArch":"linux/amd64","Os":"linux"}} root@fedora:/var/home/test# podman version --format json Error: creating lock file directory: mkdir /usr/lib/containers/storage/overlay-images: read-only file system
...and it's not just `podman version` either, `podman build` fails the same way. So it seems like this problem generalizes out to "cannot use Podman as root".
Proposing as a Beta blocker as a violation of Beta IoT criterion "The Podman container runtime must be present on all images and installed by default when using the ISO installer. It must be possible to deploy a container image" - https://fedoraproject.org/wiki/Fedora_40_Beta_Release_Criteria#Podman_container_runtime - when running as root (I'm assuming/expecting this would affect the IoT image too).
Renamed the bug as per comment 7
Oh, and it seems clear what broke this - https://src.fedoraproject.org/rpms/containers-common/c/0a797de0d4e95624fdc69c4b188c914c9076a4e2?branch=rawhide from dwalsh, which just says "local build" with no further explanation. @dwalsh ?
also, I think this is specific to atomic systems, a test which also runs podman as root but which is run on a traditional non-atomic system (Server) is passing.
CCing Timothée and Jonathan, in case this is an rpm-ostree problem.
Ok I pushed a new version. with all of the additional store content added. containers-common-0.57.1-7.fc40
Thanks Dan! In future could you produce commit messages with more accurate and descriptive titles than "local build"? :)
These messages are autogenerated. Usually I am just updating the man pages and conf files from all of the libraries. This time, I added a default director /usr/lib/containers/storage for use as an additional store. The problem is podman expects a couple of lock files to exist in the directory and will create them if they don't exist. On silverblue and other ostree OS's /usr is read/only and was preventing the creation of these lock files. With this PR I pre-create the lock files.
looks like the new build fixed it, thanks.