Bug 2481304

Ah, hmm. openQA actually has a check for this, but it looks like the text used to describe this situation changed at some point. The openQA check looks for the string "deleted to break ordering":

https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/src/commit/b9a4c6e5d99573c57e0a2c07f09d7a8d11857bc4/tests/base_services_start.pm#L13

but it looks like that's changed to this "Ordering cycle found, skipping" message. I guess I'll have to tweak the test.

The practical bug doesn't seem to have affected openQA at all...presumably for some reason systemd always resolved the cycle a different way in openQA tests, a way that didn't cause any tests to fail...I would expect skipping NM to make tests fail.

Hmm, well, seems more complicated than that. The "deleted to break ordering cycle" message is still present in the code, in src/core/transaction.c :

                if (delete) {
                        const char *status;
                        /* logging for j not k here to provide a consistent narrative */
                        log_struct(LOG_WARNING,
                                   LOG_UNIT_MESSAGE(j->unit,
                                                    "Job %s/%s deleted to break ordering cycle starting with %s/%s",
                                                    delete->unit->id, job_type_to_string(delete->type),

so...I guess maybe this doesn't happen on any package set openQA tests, or something? Can you get a journal log from an affected boot? I think the journal should have more useful messages...

Possibly related:

https://github.com/systemd/systemd/pull/40980#discussion_r2949473028

Seems maybe cloud-init will need a fix?

I will get the journal, but seems people already run into this when reviewing the new imds systemd feature

Getting the journal is not trivial, so I gave up. Please let me know if I should retry and it would be crucial to have it.

Nah, knowing it's cloud-init is likely sufficient.

I'm a bit confused here where exactly the dependency loop happens.

If you run systemd-analyze on systemd-imds-import.service, it confirms that there is a dependency loop:

# systemd-analyze verify systemd-imds-import.service
sysinit.target: Found ordering cycle: systemd-firstboot.service/start after systemd-imds-import.service/start after network-online.target/start after NetworkManager-wait-online.service/start after sysinit.target/start - after systemd-firstboot.service
sysinit.target: Job systemd-firstboot.service/start deleted to break ordering cycle starting with sysinit.target/start
sysinit.target: Found ordering cycle: systemd-imds-import.service/start after network-online.target/start after NetworkManager-wait-online.service/start after sysinit.target/start - after systemd-imds-import.service
sysinit.target: Job network-online.target/start deleted to break ordering cycle starting with sysinit.target/start

_But_, systemd-imds-import.service has ConditionPathExists=/etc/initrd-release, so it runs (or should run) only in initrd:

# systemctl start --verbose systemd-imds-import.service
May 26 07:31:44 ce6b84d4-301e-4d9f-b983-274bccdc8220 systemd[1]: systemd-imds-import.service - Import System Credentials from IMDS skipped, unmet condition check ConditionPathExists=/etc/initrd-release

But dracut initrds, at least on my machines and on my test x86_64 machine from TF, have neither cloud-init nor systemd-imds-import.service in them, so the dependency loop can't happen there either.

Miro, is this some specific setup you got there where the dependency loop can happen or is it just a "stock" Rawhide machine from Testing Farm?

Conditions are evaluated at the time the start job is run. In other words, it's enough to have the unit present and enabled to affect the boot transaction, even if it is eventually skipped. I'll submit a patch to systemd, I think the ordering in the unit should be removed.

Yeah, I just realized that as well. And the issue here is that on a IMDS-aware system the systemd-imds-generator creates a wants symlink on systemd-imds-import.service that pulls it into the transaction even on a "real" root:

# SYSTEMD_PROC_CMDLINE="systemd.imds=1" /usr/lib/systemd/system-generators/systemd-imds-generator gen/
IMDS support enabled, pulling in IMDS units.
# ls -l gen/sysinit.target.wants/
total 0
lrwxrwxrwx. 1 root root 58 May 26 07:55 systemd-imds-early-network.service -> /usr/lib/systemd/system/systemd-imds-early-network.service
lrwxrwxrwx. 1 root root 51 May 26 07:55 systemd-imds-import.service -> /usr/lib/systemd/system/systemd-imds-import.service

And that's where the loop comes from.

Stock Fedora Cloud Base image from AWS, no modifications except the update to the latest packages (tested systemd only). But I believe the latest images which already includes it has the same issue.

*** Bug 2481466 has been marked as a duplicate of this bug. ***

FEDORA-2026-d51e462be4 (systemd-261~rc2-1.fc45) has been submitted as an update to Fedora 45.
https://bodhi.fedoraproject.org/updates/FEDORA-2026-d51e462be4

FEDORA-2026-d51e462be4 (systemd-261~rc2-1.fc45) has been pushed to the Fedora 45 stable repository.
If problem still persists, please make note of it in this bug report.