Bug 2390898

Summary: auditd and sometimes other services do not run on current cloud images due to ordering cycle
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: cloud-initAssignee: Jeremy Cline <jeremy>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 43CC: dustymabe, gholms, jeremy, lars, mhayden, python-packagers-sig, robatino, rominf
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: AcceptedFreezeException
Fixed In Version: cloud-init-25.2-5.fc43 Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-08-28 19:23:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2324224, 2324225    

Description Adam Williamson 2025-08-25 19:20:31 UTC
Starting with Fedora-Rawhide-20250815.n.0, we got a lot of cases of services being deleted due to an "ordering cycle" with cloud-init-local.service:

https://openqa.fedoraproject.org/tests/3612516#step/base_services_start/7

that means those services will not start. In 20250816.n.0 most of them seemed to go away, but a couple remained, now claiming the cycle is with sysinit.target/start:

https://openqa.fedoraproject.org/tests/3630900#step/base_services_start/7

and on 20250817.n.0, only one remained, auditd:

https://openqa.fedoraproject.org/tests/3636315#step/base_services_start/7

On 20250818.n.0 we were back to a lot:

https://openqa.fedoraproject.org/tests/3640628#step/base_services_start/7

It's varied a bit day-to-day since, but *at least* auditd *always* seems to fail. All of this seems to stem back to 20250815.n.0 - and most likely to cloud-init-25.2 , which was introduced in that compose, both for Rawhide and F43.

Comment 1 Adam Williamson 2025-08-25 19:21:55 UTC
Proposing as a Final blocker - "All system services present after installation with one of the release-blocking package sets must start properly, unless they require hardware which is not present", https://fedoraproject.org/wiki/Fedora_43_Final_Release_Criteria#System_services . Also proposing as a Beta FE, I'd say we should fix this for Beta if we can.

Comment 2 Jeremy Cline 2025-08-25 20:03:29 UTC
Okay, so it looks like Fedora's systemd unit doesn't include DefaultDependencies=no[0] but it _does_ include a Before=auditd.service which itself does specify DefaultDependencies=no. So, based on my reading of the documentation, systemd adds a dependency to cloud-init-local.service of type Requires= and After= on sysinit.target, but auditd includes Before=sysinit.target.

Before=auditd.service got added in https://github.com/canonical/cloud-init/commit/cc8d1b4c464de6883bca50c0822c44fd3ba46735. I'll prep a patch to ensure Fedora has DefaultDependencies=no.

[0] https://github.com/canonical/cloud-init/blob/890873f50d7f14dc235f5eaf2820aae3288a98b1/systemd/cloud-init-local.service.tmpl#L6

Comment 3 Jeremy Cline 2025-08-25 21:39:27 UTC
Okay, I submitted https://github.com/canonical/cloud-init/pull/6423 and did a Rawhide build with it in https://bodhi.fedoraproject.org/updates/FEDORA-2025-1426ef0ad4, so we'll see how the tests do tomorrow and go from there.

Comment 4 Adam Williamson 2025-08-26 17:04:02 UTC
On a personal note, I am very happy that the check I added for this in 2018 finally proved useful. :P

https://pagure.io/fedora-qa/os-autoinst-distri-fedora/c/b4cd1b4a9e4fb25f1238843ebbb1e0f02088a2fa

Comment 5 Fedora Update System 2025-08-26 17:56:23 UTC
FEDORA-2025-39306e86fd (cloud-init-25.2-4.fc43) has been submitted as an update to Fedora 43.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-39306e86fd

Comment 6 Adam Williamson 2025-08-26 23:20:37 UTC
Unfortunately the fix does not work.

https://openqa.fedoraproject.org/tests/3668307#step/base_services_start/7

Comment 7 Fedora Update System 2025-08-27 01:00:38 UTC
FEDORA-2025-39306e86fd has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-39306e86fd`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-39306e86fd

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Jeremy Cline 2025-08-27 13:39:51 UTC
Last night I made a tweak to the proposed patch to drop the `After=dbus.socket` requirement from `cloud-init-main.service` since that pushed the service back to `After=sysinit.target`: https://bodhi.fedoraproject.org/updates/FEDORA-2025-7acc66c0d1. Since it seemed to work (thanks Adam for checking) I've updated the proposed F43 update to 25.2-5 as well.

Comment 9 Fedora Update System 2025-08-28 01:34:49 UTC
FEDORA-2025-39306e86fd has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-39306e86fd`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-39306e86fd

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Adam Williamson 2025-08-28 15:39:31 UTC
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1879 , marking accepted.

Comment 11 Fedora Update System 2025-08-28 19:23:56 UTC
FEDORA-2025-39306e86fd (cloud-init-25.2-5.fc43) has been pushed to the Fedora 43 stable repository.
If problem still persists, please make note of it in this bug report.