Bug 2350605 - The dev-gpt-auto-root.device times out and makes system not bootable.
Summary: The dev-gpt-auto-root.device times out and makes system not bootable.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 42
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F42BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2025-03-07 13:06 UTC by Lukas Ruzicka
Modified: 2025-03-12 14:57 UTC (History)
15 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2025-03-12 14:57:40 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
The error messages shown during the boot process. (4.01 MB, image/jpeg)
2025-03-07 13:09 UTC, Lukas Ruzicka
no flags Details
The final rdsosreport lines where the issue is mentioned. (4.73 MB, image/jpeg)
2025-03-07 13:10 UTC, Lukas Ruzicka
no flags Details
journalctl output (115.99 KB, text/plain)
2025-03-07 14:31 UTC, Lukas Ruzicka
no flags Details

Description Lukas Ruzicka 2025-03-07 13:06:45 UTC
On Fedora 42 Beta 1.2 non-live isos, i.e. on Everything and Server netinst, and Server DVD iso, the dev-gpt-auto-root.device times out during the installation and the system ends up in an emergency console, thus becoming not installable. 

This only happens on bare metal. In a VM, everything looks normal. 

I am attaching the error messages that I could photograph from the screen for the time's being, if more is needed, please let me know.

Reproducible: Always

Steps to Reproduce:
1. Download the Server DVD iso, for example at https://kojipkgs.fedoraproject.org/compose/42/Fedora-42-20250307.0/compose/Server/x86_64/iso/Fedora-Server-dvd-x86_64-42_Beta-1.2.iso
2. Burn it onto a flash drive and boot to install.
3. You will see the situation afterwards.
Actual Results:  
System cannot be installed from a USB media.

Expected Results:  
System should work on GPT devices just fine and should be installable.

See the screenshots attached.

Comment 1 Lukas Ruzicka 2025-03-07 13:09:53 UTC
Created attachment 2079267 [details]
The error messages shown during the boot process.

Comment 2 Lukas Ruzicka 2025-03-07 13:10:53 UTC
Created attachment 2079268 [details]
The final rdsosreport lines where the issue is mentioned.

Comment 3 Fedora Blocker Bugs Application 2025-03-07 13:12:21 UTC
Proposed as a Blocker for 42-beta by Fedora user lruzicka using the blocker tracking app because:

 Violates the boot criterion for release blocking images.

https://fedoraproject.org/wiki/Basic_Release_Criteria#Release-blocking_images_must_boot

Comment 4 Kamil Páral 2025-03-07 13:28:08 UTC
This bug only occurs when the install medium is booted in UEFI mode. With the legacy BIOS mode, the error doesn't occur.

Comment 5 Zbigniew Jędrzejewski-Szmek 2025-03-07 13:46:10 UTC
systemd-gpt-auto-generator is active when root is not specified (or is specified as root=gpt-auto) and the system uses uefi. It sounds like the image does not follow the Discoverable Partitions Specification and root is not set. Having a log from the boot would be useful.

Comment 6 Lukas Ruzicka 2025-03-07 13:50:30 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)
> systemd-gpt-auto-generator is active when root is not specified (or is
> specified as root=gpt-auto) and the system uses uefi. It sounds like the
> image does not follow the Discoverable Partitions Specification and root is
> not set. Having a log from the boot would be useful.

What file do you want to be collected?

Comment 7 Zbigniew Jędrzejewski-Szmek 2025-03-07 13:56:34 UTC
'journalctl' output.

Comment 8 Lukas Ruzicka 2025-03-07 14:31:12 UTC
Created attachment 2079271 [details]
journalctl output

Comment 9 Zbigniew Jędrzejewski-Szmek 2025-03-07 14:45:13 UTC
Mar 07 14:15:25 fedora systemd-sysusers[375]: Creating group 'systemd-journal' with GID 190.
Mar 07 14:15:25 fedora systemd[1]: Started systemd-journald.service - Journal Service.
Mar 07 14:15:25 fedora systemd-sysusers[375]: Creating group 'dbus' with GID 81.
Mar 07 14:15:25 fedora systemd-sysusers[375]: Creating user 'dbus' (System Message Bus) with UID 81 and GID 81.

That doesn't make sense. Users and groups should be precreated, since this the
initrd is a static image and creation of users requires locking and slows things
down.

Mar 07 14:15:25 fedora systemd-tmpfiles[670]: /usr/lib/tmpfiles.d/var.conf:14: Duplicate line for path "/var/log", ignoring.

Some conflicting configuration?

Mar 07 14:16:10 fedora systemd[1]: dracut-pre-udev.service: Deactivated successfully.
Mar 07 14:16:10 fedora systemd[1]: dracut-pre-udev.service: Unit process 1111 (rpcbind) remains running after unit stopped.
Mar 07 14:16:10 fedora systemd[1]: dracut-pre-udev.service: Unit process 1115 (rpc.statd) remains running after unit stopped.
Mar 07 14:16:10 fedora systemd[1]: dracut-pre-udev.service: Unit process 1120 (rpc.idmapd) remains running after unit stopped.
Mar 07 14:16:21 fedora systemd[1]: dracut-pre-udev.service: Found left-over process 1111 (rpcbind) in control group while starting unit. Ignoring.
Mar 07 14:16:21 fedora systemd[1]: dracut-pre-udev.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 07 14:16:21 fedora systemd[1]: dracut-pre-udev.service: Found left-over process 1115 (rpc.statd) in control group while starting unit. Ignoring.
Mar 07 14:16:21 fedora systemd[1]: dracut-pre-udev.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 07 14:16:21 fedora systemd[1]: dracut-pre-udev.service: Found left-over process 1120 (rpc.idmapd) in control group while starting unit. Ignoring.
Mar 07 14:16:21 fedora systemd[1]: dracut-pre-udev.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 07 14:16:22 fedora lldpad[2586]: another lldpad instance is running
Mar 07 14:16:22 fedora lldpad[2586]: failed to register client interface

Yikes. It seems that dracut is starting things not through systemd.
I don't think this is related to the current problem, but it seems
very messy.

Mar 07 14:15:25 fedora kernel: Command line: BOOT_IMAGE=/images/pxeboot/vmlinuz inst.stage2=hd:LABEL=Fedora-E-dvd-x86_64-42 quiet

Anyway, I think the issue as I suggested originally: systemd-gpt-auto-generator is
active, since there is no configuration to take priority and disable it.
But I'm not sure what changed, it's been like that for a long time.

Comment 10 Kamil Páral 2025-03-07 14:46:17 UTC
Quite interestingly, I only see this error on bare metal, but not in a VM (even though it's an UEFI VM).

Comment 11 Zbigniew Jędrzejewski-Szmek 2025-03-07 14:48:32 UTC
That's interesting. I wanted to ask about that, but then I thought I'd just test it myself. That was a few hours ago, the image is still downloading from koji :(

Comment 12 Adam Williamson 2025-03-09 17:15:44 UTC
Can we figure out what the last image that booted normally on an affected system is?

I'd look for images across systemd and maybe kernel versions - an image from before systemd 257.4, for instance.

I can't test this myself till Tuesday, as I'm away and don't have any old images handy.

Comment 13 abrouwers 2025-03-10 00:56:48 UTC
Just a data point, but I was interested in installing fedora 42 on my thinkpad, and found this bug. After encountering it with the beta-1.2 ISO, I worked backwards down this list:

https://kojipkgs.fedoraproject.org/compose/branched/

While I didn't try ALL of them (them being Everything-netinst, on x86_64), none of 0302, 0225, or 0222 would successfully boot to the installer. On each, I notice the same behavior as above - a timeout with systemd-gpt-auto-generator, and I'm dropped to an emergency shell.

Comment 14 Adam Williamson 2025-03-10 04:25:41 UTC
Thanks a lot for the info! It's helpful.

Yeah, those composes only go back a couple of weeks (for storage limitation reasons). Some of us tend to have random older images lying around, if we put our heads together we ought to be able to come up with some kind of a delta.

Comment 15 Zbigniew Jędrzejewski-Szmek 2025-03-10 09:23:03 UTC
In a VM, I see this message from systemd-gpt-auto-generator:
systemd-gpt-auto-generator[310]: EFI loader partition uknown, exiting.
systemd-gpt-auto-generator[310]: (The boot loader did not set EFI variable LoaderDevicePartUUID.)

On bare metal, grub may set LoaderDevicePartUUID.
Support for that was added in e0fa7dc84c03c7089b458137531a2913aa9e92b0:
Fri May 26 13:35:51 2023, 'bli: Add a module for the Boot Loader Interface'.

I don't know why the var would be set in one case and not the other, but
in general we should expect for it to be set. So I think we need to disable
systemd-gpt-auto-generator in the installer.

One option is to stop installing /usr/lib/systemd/system-generators/systemd-gpt-auto-generator,
a second option is to mask it via 'touch /etc/systemd/system-generators/systemd-gpt-auto-generator',
the third option is to add 'systemd.gpt_auto=0' on the kernel command line.
A fourth option would be to change the systemd-gpt-auto-generator logic
to disable itself if the root device is specified in some other way.
I tried to find some documentation for BOOT_IMAGE, but it seems that no
documentation or specification exists.

Comment 16 Zbigniew Jędrzejewski-Szmek 2025-03-10 09:25:57 UTC
Ah, no. BOOT_IMAGE is the kernel. Not relevant in this context at all.

So yeah, I think the solution is for whatever sets 'inst.stage2=' to
also add 'systemd.gpt_auto=0' on the kernel command line.

Comment 17 Kamil Páral 2025-03-10 13:11:21 UTC
Accepted as a Beta blocker:
https://pagure.io/fedora-qa/blocker-review/issue/1783

Comment 18 Kamil Páral 2025-03-10 14:39:00 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #16)
> So yeah, I think the solution is for whatever sets 'inst.stage2=' to
> also add 'systemd.gpt_auto=0' on the kernel command line.

I added 'systemd.gpt_auto=0' and it successfully boots on previously affected hardware. The boot option doesn't persist on the installed system, which seems correct (but I'm not sure if it behaves the same for images with this option baked in), and the installed system also boots fine.

So yes, the workaround to add 'systemd.gpt_auto=0' as a default boot option seems to a path forward. Now, I wonder where we can configure that...

Comment 19 Adam Williamson 2025-03-10 15:14:24 UTC
For installer images and lmc-created lives, it's lorax, I believe. For kiwi-created lives, kiwi or the templates, I think. I don't know if we need it for lives, though. CCing bcl and neal.

Comment 20 Brian Lane 2025-03-10 16:41:35 UTC
I can reproduce it with my qemu uefi+sb test script if I use the iso as -hda instead of -cdrom, simulating writing it to a USB stick.

This PR fixes it for the boot.iso:
https://github.com/weldr/lorax/pull/1453

I haven't checked the livemedia iso yet.

Comment 21 Adam Williamson 2025-03-10 22:59:51 UTC
Hah. We've just been talking (again) about getting that exact test in openQA: https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/34

Comment 22 Fedora Update System 2025-03-11 07:08:03 UTC
FEDORA-2025-9721ad93dd (lorax-42.6-1.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-9721ad93dd

Comment 23 Kamil Páral 2025-03-12 05:24:35 UTC
Unfortunately I can still see this problem with Fedora-Server-dvd-x86_64-42_Beta-1.3.iso , no change. We need to investigate whether the lorax fix didn't work, or whether the new lorax wasn't used for in image build process.

Comment 24 Fedora Update System 2025-03-12 07:07:04 UTC
FEDORA-2025-9721ad93dd (lorax-42.6-1.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 25 Kamil Páral 2025-03-12 08:01:55 UTC
Latest lorax wasn't used for the image build process. A test iso built by the new lorax seems to work fine. We're requesting a new official release candidate now.

Comment 26 Kamil Páral 2025-03-12 14:57:40 UTC
This is fixed in F42 Beta RC1.4. The lorax update is already stable, so it seems we can close this.


Note You need to log in before you can comment on or make changes to this bug.