Bug 1709944

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

I had the same issue with Fedora 31 and booting an efistub kernel. 
Startup would be stuck at "A start job is running for /dev/gpt-auto-root" and drop me to a rescue shell. I could continue to boot with the commands outlined in the original post.

I could resolve the problem by adding the "root=/dev/..." kernel command line parameter to my boot entry.

Adding root=/dev/vda1 in bootloader menu fixed it for me.
I also noticed that by default kernel commandline is very spartan: initramfs=… $kernelopts
My guess is that something is not expanding $kernelopts or generating config wrong.
I'm using systemd-boot

> I'm using systemd-boot

Oh, that's problematic. The boot-loader-spec entries that grub creates and supports are sadly
incompatible with sd-boot and the BLS specification. Among other incompatibilities, they came up
with the $kernelopts idea, but that is completely grub-specific, because (apart from sd-boot
having no concept of variable expansion), $kernelopts are set in some grub-specific config file
somewhere.

> Adding root=/dev/vda1 

The default kernel-install logic is to copy /proc/cmdline into new entries. So what I *think*
happened, is that you booted once with a broken boot entry with $kernelopts (which is obviously
not expanded by sd-boot), and when new kernels are installed, the same line is copied into them.
If you fixed that and booted into that entry, entries that are created subsequently should
"inherit" the root= argument.

> A start job is running for /dev/gpt-auto-root

That's the effect of systemd-gpt-auto-generator, which tries to automount the root
partition based on GPT partition type. It only kicks in if root= is not specified.
It seems that it is not working properly, but that is most likely an unrelated issue.
One you have the root= arg, this shouldn't matter.

After thinking about this some more, I don't think there's anything to solve here.
You don't have root= specified, so systemd-gpt-auto-generator tries to do its thing, and
it fails because you don't have the right partition types set (nothing in Fedora sets
them automatically), so finding the root fails. But systemd-gpt-auto-generator runs
before the root partition is available, so it cannot check the partition types are OK.

The mount would fail anyway, just with a different timeout and messages. We should improve
the messaging around this, I'll file a PR to do that, but it won't change anything
fundamental.

The fact that unexpanded $kernelopts was present on the kernel command line was most
likely caused by either a bug somewhere (#1712033 looks vaguely related) or a mistake
when copying the boot loader entries. I does not happen with up-to-date grub2 and
systemd, and we probably can't figure out what happened.

> without masking, systemd unmounts vda1 immediately. 

This happens because sysroot.mount has What=/dev/gpt-auto-root, so systemd says
sysroot.mount: Unit is bound to inactive unit dev-gpt\x2dauto\x2droot.device. Stopping, too.
Maybe we should be smarter, and notice that the unit is was mounted externally, and
not use the deps from the unit file then. But that is a different bug.

Yeah. The real bug is in unexpanded $kernelopts, but it's not systemd's fault.
Additionally, I thought I have GPT partition scheme with correct UUID for rootfs automounting, but it's not the case. I have legacy MBR partitions in this VM.

BTW, https://github.com/systemd/systemd/pull/14196 is the patchset I wrote while debugging this.
It doesn't change anything important, but adds messages here and there.

Oh, bugzilla.

By accident I look into 10_linux file in /etc/grub.d and there's this comment:

# The kernelopts variable should be defined in the grubenv file. But to ensure that menu
# entries populated from BootLoaderSpec files that use this variable work correctly even
# without a grubenv file, define a fallback kernelopts variable if this has not been set.
#
# The kernelopts variable in the grubenv file can be modified using the grubby tool or by
# executing the grub2-mkconfig tool. For the latter, the values of the GRUB_CMDLINE_LINUX
# and GRUB_CMDLINE_LINUX_DEFAULT options from /etc/default/grub file are used to set both
# the kernelopts variable in the grubenv file and the fallback kernelopts variable.
if [ -z "\${kernelopts}" ]; then
  set kernelopts="root=${LINUX_ROOT_DEVICE} ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}"
fi


Mine grubenv DOES NOT have kernelopts:

$ grub2-editenv list
saved_entry=Fedora (4.17.0-0.rc1.git3.1.fc29.x86_64) 29 (Server Edition)
boot_success=1

So boot snippets generator is supposed to put default kernelopts (with root=…) instead, but it isn't working.
So this looks like a GRUB error - missing kernelopts in grubenv is not handled correctly.

(In reply to Tomasz Torcz from comment #10)
> By accident I look into 10_linux file in /etc/grub.d and there's this
> comment:
> 
> # The kernelopts variable should be defined in the grubenv file. But to
> ensure that menu
> # entries populated from BootLoaderSpec files that use this variable work
> correctly even
> # without a grubenv file, define a fallback kernelopts variable if this has
> not been set.
> #
> # The kernelopts variable in the grubenv file can be modified using the
> grubby tool or by
> # executing the grub2-mkconfig tool. For the latter, the values of the
> GRUB_CMDLINE_LINUX
> # and GRUB_CMDLINE_LINUX_DEFAULT options from /etc/default/grub file are
> used to set both
> # the kernelopts variable in the grubenv file and the fallback kernelopts
> variable.
> if [ -z "\${kernelopts}" ]; then
>   set kernelopts="root=${LINUX_ROOT_DEVICE} ro ${GRUB_CMDLINE_LINUX}
> ${GRUB_CMDLINE_LINUX_DEFAULT}"
> fi
> 
> 
> Mine grubenv DOES NOT have kernelopts:
> 
> $ grub2-editenv list
> saved_entry=Fedora (4.17.0-0.rc1.git3.1.fc29.x86_64) 29 (Server Edition)
> boot_success=1
> 
> So boot snippets generator is supposed to put default kernelopts (with
> root=…) instead, but it isn't working.
> So this looks like a GRUB error - missing kernelopts in grubenv is not
> handled correctly.

There were some fixes regarding this, but that requires to re-generate your grub.cfg since that's not done on kernel upgrades.

Could you please try executing grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg ?

I did grub2-mkconfig, followed by 'dnf upgrade' which installed new kernel. Unfortunately $kernelopts is still unexpanded/not substituted with default values:

boot$ rg kernelopts
loader/entries/c7ac9fbb17429ab8a5439aae3e4ac066-5.5.0-0.rc6.git2.1.fc32.x86_64.conf
5:options $kernelopts


so boot with systemd-boot still fails.

(In reply to Tomasz Torcz from comment #12)
> I did grub2-mkconfig, followed by 'dnf upgrade' which installed new kernel.
> Unfortunately $kernelopts is still unexpanded/not substituted with default
> values:
> 
> boot$ rg kernelopts
> loader/entries/c7ac9fbb17429ab8a5439aae3e4ac066-5.5.0-0.rc6.git2.1.fc32.
> x86_64.conf
> 5:options $kernelopts
> 
> 
> so boot with systemd-boot still fails.

You are trying the use the BLS snippets created by /usr/lib/kernel/install.d/20-grub.install from sd-boot, but instead you need to create them with /usr/lib/kernel/install.d/90-loaderentry.install.

The kernel install scripts test the existence of the $ESP/<machine-id> directory as an indication for which one of these scripts should be used to generate the BLS snippets. Do you have a <machine-id> directory in your ESP? Can you please share the output of bootctl status ?

If sd-boot is installed and the directory doesn't exist, you can try to remove it and install it again, i.e (bootctl remove && bootctl install).

This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Information requested:

– my ESP partition:
/dev/sda1: UUID="9481-ADC7" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="Linux filesystem" PARTUUID="b66cc599-aad7-4a16-b7f6-6c91c4119991"

– mounted as /boot:
TARGET SOURCE    FSTYPE OPTIONS
/boot  /dev/sda1 vfat   rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro

– I have NO $ESP/<machine-id> directory

– bootctl output:
System:
     Firmware: UEFI 2.60 (EDK II 1.00)
  Secure Boot: disabled
   Setup Mode: user

Current Boot Loader:
      Product: systemd-boot v243~rc1-2.fc31
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
          ESP: /dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991
         File: └─/EFI/systemd/systemd-bootx64.efi

Random Seed:
 Passed to OS: no
 System Token: not set
       Exists: no

Available Boot Loaders on ESP:
          ESP: /boot (/dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991)
         File: └─/EFI/systemd/systemd-bootx64.efi (systemd-boot v243~rc1-2.fc31)
         File: └─/EFI/BOOT/BOOTX64.EFI (systemd-boot v243~rc1-2.fc31)

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0008
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991
         File: └─/EFI/systemd/systemd-bootx64.efi

Boot Loader Entries:
        $BOOT: /boot (/dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991)

Default Boot Loader Entry:
        title: Fedora (5.6.0-0.rc2.git1.1.fc33.x86_64) 33 (Rawhide)
           id: c7ac9fbb17429ab8a5439aae3e4ac066-5.6.0-0.rc2.git1.1.fc33.x86_64.conf
       source: /boot/loader/entries/c7ac9fbb17429ab8a5439aae3e4ac066-5.6.0-0.rc2.git1.1.fc33.x86_64.conf
      version: 5.6.0-0.rc2.git1.1.fc33.x86_64
        linux: /vmlinuz-5.6.0-0.rc2.git1.1.fc33.x86_64
       initrd: /initramfs-5.6.0-0.rc2.git1.1.fc33.x86_64.img
      options: $kernelopts

– at this point I made changes:
  – created <machineid> directory on rootdir of ESP
  – upgraded to new kernel
  – rebooted

– reboot was SUCCESSFUL

– bootctl after creating the machineid dir:
System:
     Firmware: UEFI 2.60 (EDK II 1.00)
  Secure Boot: disabled
   Setup Mode: user

Current Boot Loader:
      Product: systemd-boot v243~rc1-2.fc31
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
          ESP: /dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991
         File: └─/EFI/systemd/systemd-bootx64.efi

Random Seed:
 Passed to OS: no
 System Token: not set
       Exists: no

Available Boot Loaders on ESP:
          ESP: /boot (/dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991)
         File: └─/EFI/systemd/systemd-bootx64.efi (systemd-boot v243~rc1-2.fc31)
         File: └─/EFI/BOOT/BOOTX64.EFI (systemd-boot v243~rc1-2.fc31)

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0008
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991
         File: └─/EFI/systemd/systemd-bootx64.efi

Boot Loader Entries:
        $BOOT: /boot (/dev/disk/by-partuuid/b66cc599-aad7-4a16-b7f6-6c91c4119991)

Default Boot Loader Entry:
        title: Fedora 33 (Server Edition)
           id: c7ac9fbb17429ab8a5439aae3e4ac066-5.6.0-0.rc2.git3.1.fc33.x86_64.conf
       source: /boot/loader/entries/c7ac9fbb17429ab8a5439aae3e4ac066-5.6.0-0.rc2.git3.1.fc33.x86_64.conf
      version: 5.6.0-0.rc2.git3.1.fc33.x86_64
   machine-id: c7ac9fbb17429ab8a5439aae3e4ac066
        linux: /c7ac9fbb17429ab8a5439aae3e4ac066/5.6.0-0.rc2.git3.1.fc33.x86_64/linux
       initrd: /c7ac9fbb17429ab8a5439aae3e4ac066/5.6.0-0.rc2.git3.1.fc33.x86_64/initrd
      options: $kernelopts root=/dev/vda1


So my issue is fixed, missing $ESP/<machine-id> directory was the culprit. Thank you for investigation.

Can we have this caveat documented somewhere? This machinery is quite complicated and it's not clear that existence of machineid directory changes behaviour.

This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.

Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Hello.

I know this issue is formally EOL and that my own issue isn't exactly the same than this one (because I don’t want to boot a “regular” root partition: I want to boot Anaconda installer) but I haven't found any place more suitable to expose my problem, which could be interesting to another people too. 


I want to create a bootable USB pendrive using the Fedora Server 34 iso and I want to do “manually”. So I’ve created two partitions in my pendrive (/dev/sdb): the first one, /dev/sdb1, is the ESP (where I’ll put the systemd-boot bootloader) and the second one, /dev/sdb2, is formatted in Ext4 (where I’ll put all the iso content) and I’ve put the Fedora Server 34 iso in my CD drive /dev/sr0. And then I’ve done this from my running system:

e2label /dev/sdb2 manolo
mkdir /mnt/cd /mnt/esp /mnt/iso
mount /dev/sr0 /mnt/cd
mount /dev/sdb1 /mnt/esp
mount /dev/sdb2 /mnt/iso
cp -r /mnt/cd/\* /mnt/iso
bootctl --esp-path=/mnt/esp --no-variables install
cp /mnt/iso/images/pxeboot/vmlinuz /mnt/esp/EFI/systemd
cp /mnt/iso/images/pxeboot/initrd.img /mnt/esp/EFI/systemd
echo "default pepe" > /mnt/esp/loader/loader.conf
echo "timeout 5" >> /mnt/esp/loader/loader.conf
echo "title Arrencar LiveCD Fedora" > /mnt/esp/loader/entries/pepe.conf
echo "linux /EFI/systemd/vmlinuz" >> /mnt/esp/loader/entries/pepe.conf
echo "initrd /EFI/systemd/initrd.img" >> /mnt/esp/loader/entries/pepe.conf
echo "options inst.stage2=hd:LABEL=manolo" >> /mnt/esp/loader/entries/pepe.conf

When I boot from my pendrive it seems kernel boots ok until I get this message: “A start job is running for /dev/gpt-auto-root” and I get stuck in a dracut shell. Notice I’ve put exactly the same kernel parameter there is in /mnt/iso/EFI/BOOT/grub.cfg, which (I suspect!) it’s the default boot loader’s configuration file used by Fedora Server iso when booting to Anaconda. So I don’t know which kernel parameter I’m missing!

Thanks a lot

PD: I’ve open this same question in Ask Fedora (https://ask.fedoraproject.org/t/stuck-while-booting-the-fedora-server-34-iso-from-an-usb-pendrive/17443) with no luck (yet)

q2dg: an old bug is definitely not the best place to ask such questions.
Unfortunately I don't know anything about the internals of our livecds, so I can't help you.

Well, I've solved (kind of).
I've put the Linux x86_64 GUID to /dev/sdb2 and that's all.
I mean, I've run fdisk /dev/sdb and, inside its shell, I've choosen "t"->"2"->"23"->"w" (number 23 is the identifier for desired GUID, 4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709)
And that's all: now Anaconda boots.
But this behaviour isn't documented anywhere and I think it's a bit tricky: someone should be study this with more care.
Thanks anyway.