Description of problem: Attempting to PXE boot with grub2 on ppc64le fails with: ``` Welcome to Open Firmware Copyright (c) 2004, 2017 IBM Corporation All rights reserved. This program and the accompanying materials are made available under the terms of the BSD License available at http://www.opensource.org/licenses/bsd-license.php Trying to load: from: /pci@800000020000000/scsi@3 ... E3404: Not a bootable device! Trying to load: from: /pci@800000020000000/ethernet@2 ... Initializing NIC Reading MAC address from device: 52:54:00:12:34:56 Requesting information via DHCP: done Using IPv4 address: 192.168.76.9 Requesting file "/boot/grub2/powerpc-ieee1275/core.elf" via TFTP from 192.168.76.2 Receiving data: 318 KBytes TFTP: Received /boot/grub2/powerpc-ieee1275/core.elf (318 KBytes) Successfully loaded Welcome to GRUB! error: ../../grub-core/script/function.c:119:can't find command `source'. ``` Version-Release number of selected component (if applicable): This started failing on the grub2-2.06-75.fc37 to grub2-2.06-84.fc37 transition. How reproducible: Always Steps to Reproduce: The code we have that sets this up is at https://github.com/coreos/coreos-assembler/blob/f502808d8e5e27f3c17d88b59033321ee18d76fc/mantle/platform/metal.go#L410-L435 But ultimately it comes down to something like `grub2-mknetdir --net-directory=/path` then creating a grub.cfg file and PXE booting that. Actual results: Failure with `error: ../../grub-core/script/function.c:119:can't find command `source'.` Expected results: Booted system Additional info: This issue was found in the Fedora CoreOS pipeline and originally opened as an issue there: https://github.com/coreos/coreos-assembler/issues/3370 We have reverted to the last known working version of GRUB until this issue is resolved: https://github.com/coreos/coreos-assembler/pull/3371
> This started failing on the grub2-2.06-75.fc37 to grub2-2.06-84.fc37 transition. Can you bisect that further? There're nine builds in there. Or, more helpfully, can you do this on rawhide builds and give me a last known good one there? On the PXE server, is there a difference in what files are fetched between working and non-working? Can you show the contents of grub.cfg?
grub2-2.06-76.fc38 is good grub2-2.06-77.fc38 is bad contents of the grub.cfg look like: ``` $ cat /var/tmp/mantle-pxe3035618054/tftp/boot/grub2/grub.cfg default=0 timeout=1 menuentry "CoreOS (BIOS/UEFI)" { echo "Loading kernel" linux /fedora-coreos-37.20230227.dev.0-live-kernel-ppc64le rd.neednet=1 ip=dhcp ignition.firstboot ignition.platform.id=metal console=hvc0 ignition.config.url=http://192.168.76.2:42869/pxe-live.ign coreos.inst.install_dev=/dev/vda coreos.inst.ignition_url=http://192.168.76.2:42869/config.ign coreos.inst.image_url=http://192.168.76.2:42869/fedora-coreos-37.20230227.dev.0-metal.ppc64le.raw coreos.inst.insecure coreos.live.rootfs_url=http://192.168.76.2:42869/fedora-coreos-37.20230227.dev.0-live-rootfs.ppc64le.img echo "Loading initrd" initrd fedora-coreos-37.20230227.dev.0-live-initramfs.ppc64le.img } ```
Thanks. Would still like to know: > On the PXE server, is there a difference in what files are fetched between working and non-working?
(In reply to Robbie Harwood from comment #3) > Thanks. > > Would still like to know: > > > On the PXE server, is there a difference in what files are fetched between working and non-working? I'll try to answer this by saying that nothing changed between a successful and failed test except for the version of GRUB2 rpm in the system. Does that help?
Not really, no. In order to network boot, configuration, payload, etc. have to be fetched over the network. Depending on setup, that'll be http or tftp; the usual servers for those will log requests. (Some are expected to fail as grub does probing.) I haven't yet managed to replicate the failure you're seeing.
I can reproduce the bug, and confirm that grub2-2.06-76.fc38 works, and grub2-2.06-77.fc38 does not.
It's also possible to reproduce using x-platform virt. You need to be running dhcp & tftp over ipv4. Install tested grub version and then grub2-mknetdir --net-directory=/var/lib/tftpboot # cat /etc/dhcp/dhcpd.conf # # DHCP Server Configuration file. # see /usr/share/doc/dhcp-server/dhcpd.conf.example # see dhcpd.conf(5) man page # option architecture-type code 93 = unsigned integer 16; subnet 192.168.124.0 netmask 255.255.255.0 { option routers 192.168.124.1; option domain-name-servers 192.168.124.1; range 192.168.124.100 192.168.124.200; next-server 192.168.124.2; filename "/boot/grub2/powerpc-ieee1275/core.elf"; } # cat /var/lib/tftpboot/boot/grub2/grub.cfg set timeout=5 menuentry 'install fedora' { linux /images/fedora/vmlinuz ip=dhcp inst.repo=http://download.eng.bos.redhat.com/released/fedora/F-37/RC/1.7/Server/ppc64le/os/ console=ttyS0 initrd /images/fedora/initrd.img } # cat /var/lib/tftpboot/boot/grub2/powerpc-ieee1275/grub.cfg source boot/grub2/grub.cfg (this one is created by the grub2-mknetdir command)
This is still an issue with the latest grub2 in stable f38 repos (grub2-2.06-95.fc38). (In reply to Robbie Harwood from comment #5) > Not really, no. > > In order to network boot, configuration, payload, etc. have to be fetched > over the network. Depending on setup, that'll be http or tftp; the usual > servers for those will log requests. (Some are expected to fail as grub > does probing.) > > I haven't yet managed to replicate the failure you're seeing. Not sure if this helps, but here's the output between a working (grub2-2.06-76.fc38) run: Trying to load: from: /pci@800000020000000/scsi@3 ... E3404: Not a bootable device! Trying to load: from: /pci@800000020000000/ethernet@2 ... Initializing NIC Reading MAC address from device: 52:54:00:12:34:56 Requesting information via DHCP: done Using IPv4 address: 192.168.76.9 Requesting file "/boot/grub2/powerpc-ieee1275/core.elf" via TFTP from 192.168.76.2 Receiving data: 318 KBytes TFTP: Received /boot/grub2/powerpc-ieee1275/core.elf (318 KBytes) Successfully loaded Welcome to GRUB! And a non-working run (grub2-2.06-95.fc38): Trying to load: from: /pci@800000020000000/scsi@3 ... E3404: Not a bootable device! Trying to load: from: /pci@800000020000000/ethernet@2 ... Initializing NIC Reading MAC address from device: 52:54:00:12:34:56 Requesting information via DHCP: done Using IPv4 address: 192.168.76.9 Requesting file "/boot/grub2/powerpc-ieee1275/core.elf" via TFTP from 192.168.76.2 Receiving data: 321 KBytes TFTP: Received /boot/grub2/powerpc-ieee1275/core.elf (321 KBytes) Successfully loaded Welcome to GRUB! error: ../../grub-core/script/function.c:119:can't find command `source'. Not much difference between the two it seems. Is there maybe a debug build of grub2 you could provide that would output more information? For server-side logs, we're using the built-in qemu tftp server. It doesn't look offhand like there's a way to have it output that information so we'd have to stand up a standalone TFTP/HTTP server instead and tweak our harness.
As Marta pointed out, https://bugzilla.redhat.com/show_bug.cgi?id=2209435 looks very similar to this - however I'm not seeing an error message, just "Welcome to GRUB!" and then a grub shell (instead of the boot menu as expected). openQA started hitting this right when the grub version updated, but I only got around to looking into it now :/ If it helps, I can dump out the whole /var/lib/tftpboot directory from a working and a failed case...
FEDORA-2024-53d986312e has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-53d986312e
FEDORA-2024-633dc7e183 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2024-633dc7e183
FEDORA-2024-633dc7e183 has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-633dc7e183` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-633dc7e183 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2024-53d986312e has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-53d986312e` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-53d986312e See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2024-53d986312e has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2024-633dc7e183 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.