Bug 2481466

Summary: systemd-imds-early-network.service fails on AWS EC2 - no IMDS auto-detection
Product: [Fedora] Fedora Reporter: Petr Sklenar <psklenar>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rawhideCC: daan.j.demeyer, fedoraproject, frantisek, lnykryn, msekleta, suraj.ghimire7, systemd-maint, vmihalko, yuwatana, zbyszek
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-261~rc2-1.fc45 Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2026-05-26 22:50:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
journalctl none

Description Petr Sklenar 2026-05-26 14:20:34 UTC
Description of problem:
The systemd-imds-early-network.service fails on AWS EC2 instances with error "No IMDS endpoint information provided or detected, cannot operate." The service expects systemd credentials (imds.vendor, imds.token_header_name, etc.) to be provided by bootloader/initrd, but these are not available on AWS EC2. The IMDS endpoint (http://169.254.169.254/) is accessible and the system correctly detects "Amazon EC2" via DMI (/sys/class/dmi/id/sys_vendor), but systemd-imdsd does not auto-detect the cloud platform.

Version-Release number of selected component:
systemd-261~rc1-1.fc45.x86_64
Fedora Linux 45 (Cloud Edition Prerelease)
Kernel: 7.1.0-0.rc5.35.fc45.x86_64

How reproducible:
Always (100%)

Steps to Reproduce:
1. Launch Fedora 45 Cloud Edition on AWS EC2 (tested on c6a.large)
2. Boot the system
3. Run: systemctl status systemd-imds-early-network.service

Actual Results:
Service fails with:
```
× systemd-imds-early-network.service - Enable Pre-IMDS Networking
     Active: failed (Result: exit-code)
   Main PID: 473 (code=exited, status=1/FAILURE)

May 26 14:08:42 systemd-imdsd[473]: Failed to read credential 'imds.vendor', ignoring: No such device or address
May 26 14:08:42 systemd-imdsd[473]: Failed to read credential 'imds.data_url', ignoring: No such device or address
[...60+ similar credential failures...]
May 26 14:08:42 systemd-imdsd[473]: No IMDS endpoint information provided or detected, cannot operate.
May 26 14:08:42 systemd[1]: systemd-imds-early-network.service: Failed with result 'exit-code'.
```

Expected Results:
Service should auto-detect AWS EC2 and successfully configure IMDS networking, OR gracefully skip if not needed without showing as failed.

Additional info:
- IMDS endpoint IS accessible: curl -s http://169.254.169.254/latest/meta-data/ works correctly
- DMI detection works: /sys/class/dmi/id/sys_vendor shows "Amazon EC2"
- Instance type: c6a.large
- systemctl --all --failed shows this as the only failed service

Root cause analysis:
systemd-imdsd requires systemd credentials to be set (imds.vendor, imds.token_header_name, imds.data_url, etc.) but:
1. No component in the boot chain sets these credentials on AWS EC2
2. systemd-imdsd lacks fallback auto-detection for common cloud platforms
3. The service is marked as "enabled-runtime" but has no mechanism to detect if it's actually needed

Suggested fixes:
1. Add auto-detection for AWS/Azure/GCP when credentials are missing (check DMI vendor string)
2. Have cloud-init or systemd-sysext provide credentials on cloud platforms
3. Change service to not fail if IMDS is genuinely unavailable (ConditionVirtualization= or similar)

Impact:
Visual clutter in systemctl --failed, but no functional impact observed - networking works correctly.

Comment 1 Petr Sklenar 2026-05-26 14:23:22 UTC
to reproduce:
https://fedoraproject.org/wiki/Test_Results:Fedora_45_Rawhide_20260523.n.0_Cloud#Amazon_EC2_(via_Testing_Farm)

with public api key:
testing-farm reserve --compose Fedora-Cloud-Base-AmazonEC2.x86_64-Rawhide-20260523.0  --arch x86_64

Comment 3 Frantisek Sumsal 2026-05-26 14:45:12 UTC
This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2481466 and is already being resolved in upstream by https://github.com/systemd/systemd/pull/42308.

*** This bug has been marked as a duplicate of bug 2481304 ***

Comment 4 Frantisek Sumsal 2026-05-26 14:50:08 UTC
On second thought, the fail is not about an ordering cycle (at least from the snippet you sent) like the other issue I mentioned. Could you, please, provide a full journal from the machine where the unit fails?

Comment 5 Petr Sklenar 2026-05-26 15:08:29 UTC
Created attachment 2142962 [details]
journalctl

Comment 6 Petr Sklenar 2026-05-26 15:09:50 UTC
Comment on attachment 2142962 [details]
journalctl

May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd-imdsd[476]: Failed to read credential 'imds.vendor', ignoring: No such device or address

Comment 7 Frantisek Sumsal 2026-05-26 15:26:00 UTC
So, there _is_ the ordering cycle from https://bugzilla.redhat.com/show_bug.cgi?id=2481304 in the journal:

May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: sysinit.target: Found ordering cycle: systemd-imds-import.service/start after network-online.target/start after cloud-init-network.service/start after basic.target/start after sockets.target/start after sssd-kcm.socket/start after sysinit.target/start - after systemd-imds-import.service
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: sysinit.target: Job systemd-imds-import.service/start deleted to break ordering cycle starting with sysinit.target/start

But the fail in systemd-imds-early-network.service is most likely a different issue:

...
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd-imdsd[476]: Failed to read credential 'imds.key_userdata', ignoring: No such device or address
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd-imdsd[476]: Failed to read credential 'imds.key_userdata_base', ignoring: No such device or address
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd-imdsd[476]: Failed to read credential 'imds.key_userdata_base64', ignoring: No such device or address
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd-imdsd[476]: No IMDS endpoint information provided or detected, cannot operate.
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: Finished lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: systemd-imds-early-network.service: Main process exited, code=exited, status=1/FAILURE
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: systemd-imds-early-network.service: Failed with result 'exit-code'.
May 26 15:04:06 ip-172-31-17-43.us-east-2.compute.internal systemd[1]: Failed to start systemd-imds-early-network.service - Enable Pre-IMDS Networking.
...

Comment 8 Zbigniew Jędrzejewski-Szmek 2026-05-26 16:28:57 UTC
https://github.com/systemd/systemd/pull/42318 is untested, but I think it should DTRT.

Comment 9 Fedora Update System 2026-05-26 21:34:18 UTC
FEDORA-2026-d51e462be4 (systemd-261~rc2-1.fc45) has been submitted as an update to Fedora 45.
https://bodhi.fedoraproject.org/updates/FEDORA-2026-d51e462be4

Comment 10 Fedora Update System 2026-05-26 22:50:28 UTC
FEDORA-2026-d51e462be4 (systemd-261~rc2-1.fc45) has been pushed to the Fedora 45 stable repository.
If problem still persists, please make note of it in this bug report.