Fedora Account System
Red Hat Associate
Red Hat Customer
After investigating instabilities in Fedora Rawhide testing in Testing Farm (Fedora CI), we found out that after updating to systemd-261~rc1-1.fc45 the systemd does not always boot successfully. The issue can be seen on console log of the instance: [[0;1;31m SKIP [0m] Ordering cycle found, skipping [0;1;39msockets.target[0m [[0;1;31m SKIP [0m] Ordering cycle found, skipping [0;1;39mNetworkManager.service[0m [[0;1;31m SKIP [0m] Ordering cycle found, skipping [0;1;39msystemd-imds- import.service[0m Previous systemd version (260.1-2.fc45) works fine, I tried 10 reboots in a row. Reproducible: Always Steps to Reproduce: 1. Boot a Fedora Rawhide cloud base image with broken systemd package or update to it and reboot 2. Machine does not always boot 3. Console log shows ordering issues causing NM to not start Actual Results: Boot ordering issues Expected Results: No boot ordering issues
Ah, hmm. openQA actually has a check for this, but it looks like the text used to describe this situation changed at some point. The openQA check looks for the string "deleted to break ordering": https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/src/commit/b9a4c6e5d99573c57e0a2c07f09d7a8d11857bc4/tests/base_services_start.pm#L13 but it looks like that's changed to this "Ordering cycle found, skipping" message. I guess I'll have to tweak the test. The practical bug doesn't seem to have affected openQA at all...presumably for some reason systemd always resolved the cycle a different way in openQA tests, a way that didn't cause any tests to fail...I would expect skipping NM to make tests fail.
Hmm, well, seems more complicated than that. The "deleted to break ordering cycle" message is still present in the code, in src/core/transaction.c : if (delete) { const char *status; /* logging for j not k here to provide a consistent narrative */ log_struct(LOG_WARNING, LOG_UNIT_MESSAGE(j->unit, "Job %s/%s deleted to break ordering cycle starting with %s/%s", delete->unit->id, job_type_to_string(delete->type), so...I guess maybe this doesn't happen on any package set openQA tests, or something? Can you get a journal log from an affected boot? I think the journal should have more useful messages...
Possibly related: https://github.com/systemd/systemd/pull/40980#discussion_r2949473028 Seems maybe cloud-init will need a fix?
I will get the journal, but seems people already run into this when reviewing the new imds systemd feature
Getting the journal is not trivial, so I gave up. Please let me know if I should retry and it would be crucial to have it.
Nah, knowing it's cloud-init is likely sufficient.
I'm a bit confused here where exactly the dependency loop happens. If you run systemd-analyze on systemd-imds-import.service, it confirms that there is a dependency loop: # systemd-analyze verify systemd-imds-import.service sysinit.target: Found ordering cycle: systemd-firstboot.service/start after systemd-imds-import.service/start after network-online.target/start after NetworkManager-wait-online.service/start after sysinit.target/start - after systemd-firstboot.service sysinit.target: Job systemd-firstboot.service/start deleted to break ordering cycle starting with sysinit.target/start sysinit.target: Found ordering cycle: systemd-imds-import.service/start after network-online.target/start after NetworkManager-wait-online.service/start after sysinit.target/start - after systemd-imds-import.service sysinit.target: Job network-online.target/start deleted to break ordering cycle starting with sysinit.target/start _But_, systemd-imds-import.service has ConditionPathExists=/etc/initrd-release, so it runs (or should run) only in initrd: # systemctl start --verbose systemd-imds-import.service May 26 07:31:44 ce6b84d4-301e-4d9f-b983-274bccdc8220 systemd[1]: systemd-imds-import.service - Import System Credentials from IMDS skipped, unmet condition check ConditionPathExists=/etc/initrd-release But dracut initrds, at least on my machines and on my test x86_64 machine from TF, have neither cloud-init nor systemd-imds-import.service in them, so the dependency loop can't happen there either. Miro, is this some specific setup you got there where the dependency loop can happen or is it just a "stock" Rawhide machine from Testing Farm?
Conditions are evaluated at the time the start job is run. In other words, it's enough to have the unit present and enabled to affect the boot transaction, even if it is eventually skipped. I'll submit a patch to systemd, I think the ordering in the unit should be removed.
Yeah, I just realized that as well. And the issue here is that on a IMDS-aware system the systemd-imds-generator creates a wants symlink on systemd-imds-import.service that pulls it into the transaction even on a "real" root: # SYSTEMD_PROC_CMDLINE="systemd.imds=1" /usr/lib/systemd/system-generators/systemd-imds-generator gen/ IMDS support enabled, pulling in IMDS units. # ls -l gen/sysinit.target.wants/ total 0 lrwxrwxrwx. 1 root root 58 May 26 07:55 systemd-imds-early-network.service -> /usr/lib/systemd/system/systemd-imds-early-network.service lrwxrwxrwx. 1 root root 51 May 26 07:55 systemd-imds-import.service -> /usr/lib/systemd/system/systemd-imds-import.service And that's where the loop comes from.
Stock Fedora Cloud Base image from AWS, no modifications except the update to the latest packages (tested systemd only). But I believe the latest images which already includes it has the same issue.
*** Bug 2481466 has been marked as a duplicate of this bug. ***
FEDORA-2026-d51e462be4 (systemd-261~rc2-1.fc45) has been submitted as an update to Fedora 45. https://bodhi.fedoraproject.org/updates/FEDORA-2026-d51e462be4
FEDORA-2026-d51e462be4 (systemd-261~rc2-1.fc45) has been pushed to the Fedora 45 stable repository. If problem still persists, please make note of it in this bug report.