Hide Forgot
Description of problem: Recent installations of Fedora 37 IoT have a long list of failing services: systemctl --all --failed UNIT LOAD ACTIVE SUB DESCRIPTION > ● dbus-broker.service loaded failed failed D-Bus System Message> ● greenboot-grub2-set-counter.service loaded failed failed Set grub2 boot count> ● greenboot-healthcheck.service loaded failed failed greenboot Health Che> ● NetworkManager.service loaded failed failed Network Manager ● polkit.service loaded failed failed Authorization Manager ● redboot-auto-reboot.service loaded failed failed Reboot on red boot s> ● rpm-ostreed.service loaded failed failed rpm-ostree System Ma> ● systemd-oomd.service loaded failed failed Userspace Out-Of-Mem> ● systemd-resolved.service loaded failed failed Network Name Resolut> ● systemd-userdbd.service loaded failed failed User Database Manage> ● systemd-oomd.socket loaded failed failed Userspace Out-Of-Mem> ● systemd-userdbd.socket loaded failed failed User Database Manage> mount | grep /sysroot /dev/mapper/fedora--iot_fedora-root on /sysroot type ext4 (ro,relatime,seclabel) Remounting /sysroot as RW I can start all services again, this also does not affect upgrades from F36 where sysroot remains RW. Version-Release number of selected component (if applicable): rpm-ostree-2022.12-4.fc37.aarch64 ostree-2022.5-2.fc37.aarch64 anaconda-37.12.1-1.fc37 How reproducible: Everytime
Proposed as a Blocker for 37-final by Fedora user coremodule using the blocker tracking app because: Proposing as an F37 blocker as it appears to violate the following criterion: All system services present after installation with one of the release-blocking package sets must start properly, unless they require hardware which is not present. https://fedoraproject.org/wiki/Fedora_37_Final_Release_Criteria#System_services
yeah, openQA has been hitting this for days. I mentioned it in IRC but didn't see any followup at the time. This prevents all the openQA tests from working, so it might be a Beta blocker, really. Let's throw it on that list for now and I'll see if it's a case of "beta functionality really doesn't work" or just "the noise throws openQA off".
This sounds like it could be a side effect of https://fedoraproject.org/wiki/Changes/Silverblue_Kinoite_readonly_sysroot Setting to block 2060976, the tracker for that Change.
Well, the fact that the change affects IoT isn't a side effect, it's in the description: "This change applies to new and existing installations of Fedora Silverblue and Kinoite and only to new installations of Fedora IoT." But the fact that it breaks everything is a problem, yeah. :D
Do you have `rw` on the kernel command line?
Yeah, it does have that there. I didn't put it there, though. It's like that out of the "box" (the IoT dvd-ostree install image, in my case). Just do a fresh install of https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-37-20220825.0/compose/IoT/x86_64/iso/Fedora-IoT-ostree-x86_64-37-20220825.0.iso without doing anything unusual, boot it, and you have 'rw' in cmdline and hit this bug. If I take that out of the cmdline or change it to 'ro', boot loops with ostree-prepare-root.service failing. openQA tests fail because they try to switch to a different VT, and there is no console running on any VT besides 1 (probably the services that should run one fail to start). This is a clear violation of Beta criterion "A system installed without a graphical package set must boot to a working login prompt without any unintended user intervention, and all virtual consoles intended to provide a working login prompt must do so.", so that supports the Beta blocker nomination.
Yeah, if I look at logs after switching to VT2 and back to VT1, I see "Failed to start autovt: Transport endpoint is not connected".
Try setting `tmp-is-dir: true` in the manifest in https://pagure.io/fedora-iot/ostree/blob/main/f/fedora-iot-base.json
Peter, can you try Colin's suggestion?
Pushed for rawhide and kicked off a compose
That looks good for Rawhide, only two failed tests now: https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=38&build=Fedora-IoT-38-20220827.0&groupid=1 I'll look into those. We'll need the change for F37 too.
Remaining failed tests both look to be caused by https://bugzilla.redhat.com/show_bug.cgi?id=2121944 .
Applied to F-37 too, thanks Adam and Colin.
confirmed, we got an F37 compose that is in the same state as Rawhide (most things work, https://bugzilla.redhat.com/show_bug.cgi?id=2121944 is an issue).