Bug 2173015

> This started failing on the grub2-2.06-75.fc37 to grub2-2.06-84.fc37 transition.

Can you bisect that further?  There're nine builds in there.  Or, more helpfully, can you do this on rawhide builds and give me a last known good one there?

On the PXE server, is there a difference in what files are fetched between working and non-working?

Can you show the contents of grub.cfg?

grub2-2.06-76.fc38 is good
grub2-2.06-77.fc38 is bad

contents of the grub.cfg look like:

```
$ cat /var/tmp/mantle-pxe3035618054/tftp/boot/grub2/grub.cfg
default=0
timeout=1
menuentry "CoreOS (BIOS/UEFI)" {
        echo "Loading kernel"
        linux /fedora-coreos-37.20230227.dev.0-live-kernel-ppc64le rd.neednet=1 ip=dhcp ignition.firstboot ignition.platform.id=metal console=hvc0 ignition.config.url=http://192.168.76.2:42869/pxe-live.ign coreos.inst.install_dev=/dev/vda coreos.inst.ignition_url=http://192.168.76.2:42869/config.ign coreos.inst.image_url=http://192.168.76.2:42869/fedora-coreos-37.20230227.dev.0-metal.ppc64le.raw coreos.inst.insecure coreos.live.rootfs_url=http://192.168.76.2:42869/fedora-coreos-37.20230227.dev.0-live-rootfs.ppc64le.img
        echo "Loading initrd"
        initrd fedora-coreos-37.20230227.dev.0-live-initramfs.ppc64le.img
}
```

Thanks.

Would still like to know:

> On the PXE server, is there a difference in what files are fetched between working and non-working?

(In reply to Robbie Harwood from comment #3)
> Thanks.
> 
> Would still like to know:
> 
> > On the PXE server, is there a difference in what files are fetched between working and non-working?

I'll try to answer this by saying that nothing changed between a successful and failed test except for the version of GRUB2 rpm in the system.

Does that help?

Not really, no.

In order to network boot, configuration, payload, etc. have to be fetched over the network.  Depending on setup, that'll be http or tftp; the usual servers for those will log requests.  (Some are expected to fail as grub does probing.)

I haven't yet managed to replicate the failure you're seeing.

I can reproduce the bug, and confirm that grub2-2.06-76.fc38 works, and grub2-2.06-77.fc38 does not.

It's also possible to reproduce using x-platform virt.
You need to be running dhcp & tftp over ipv4.
Install tested grub version and then grub2-mknetdir --net-directory=/var/lib/tftpboot

# cat /etc/dhcp/dhcpd.conf 
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp-server/dhcpd.conf.example
#   see dhcpd.conf(5) man page
#
option architecture-type code 93 = unsigned integer 16;
 
subnet 192.168.124.0 netmask 255.255.255.0 {
  option routers 192.168.124.1;
  option domain-name-servers 192.168.124.1;
  range 192.168.124.100 192.168.124.200;
    next-server 192.168.124.2; 
    filename "/boot/grub2/powerpc-ieee1275/core.elf";
}

# cat /var/lib/tftpboot/boot/grub2/grub.cfg 
set timeout=5
menuentry 'install fedora' {
  linux /images/fedora/vmlinuz ip=dhcp inst.repo=http://download.eng.bos.redhat.com/released/fedora/F-37/RC/1.7/Server/ppc64le/os/ console=ttyS0
  initrd /images/fedora/initrd.img
}

# cat /var/lib/tftpboot/boot/grub2/powerpc-ieee1275/grub.cfg 
source boot/grub2/grub.cfg
(this one is created by the grub2-mknetdir command)

This is still an issue with the latest grub2 in stable f38 repos (grub2-2.06-95.fc38).

(In reply to Robbie Harwood from comment #5)
> Not really, no.
> 
> In order to network boot, configuration, payload, etc. have to be fetched
> over the network.  Depending on setup, that'll be http or tftp; the usual
> servers for those will log requests.  (Some are expected to fail as grub
> does probing.)
> 
> I haven't yet managed to replicate the failure you're seeing.

Not sure if this helps, but here's the output between a working (grub2-2.06-76.fc38) run:

Trying to load:  from: /pci@800000020000000/scsi@3 ...
E3404: Not a bootable device!
Trying to load:  from: /pci@800000020000000/ethernet@2 ...
 Initializing NIC
  Reading MAC address from device: 52:54:00:12:34:56
  Requesting information via DHCP: done
  Using IPv4 address: 192.168.76.9
  Requesting file "/boot/grub2/powerpc-ieee1275/core.elf" via TFTP from 192.168.76.2
  Receiving data:  318 KBytes
  TFTP: Received /boot/grub2/powerpc-ieee1275/core.elf (318 KBytes)
  Successfully loaded
Welcome to GRUB!

And a non-working run (grub2-2.06-95.fc38):

Trying to load:  from: /pci@800000020000000/scsi@3 ...
E3404: Not a bootable device!
Trying to load:  from: /pci@800000020000000/ethernet@2 ...
 Initializing NIC
  Reading MAC address from device: 52:54:00:12:34:56
  Requesting information via DHCP: done
  Using IPv4 address: 192.168.76.9
  Requesting file "/boot/grub2/powerpc-ieee1275/core.elf" via TFTP from 192.168.76.2
  Receiving data:  321 KBytes
  TFTP: Received /boot/grub2/powerpc-ieee1275/core.elf (321 KBytes)
  Successfully loaded
Welcome to GRUB!

error: ../../grub-core/script/function.c:119:can't find command `source'.

Not much difference between the two it seems. Is there maybe a debug build of grub2 you could provide that would output more information?

For server-side logs, we're using the built-in qemu tftp server. It doesn't look offhand like there's a way to have it output that information so we'd have to stand up a standalone TFTP/HTTP server instead and tweak our harness.

As Marta pointed out, https://bugzilla.redhat.com/show_bug.cgi?id=2209435 looks very similar to this - however I'm not seeing an error message, just "Welcome to GRUB!" and then a grub shell (instead of the boot menu as expected). openQA started hitting this right when the grub version updated, but I only got around to looking into it now :/

If it helps, I can dump out the whole /var/lib/tftpboot directory from a working and a failed case...

FEDORA-2024-53d986312e has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-53d986312e

FEDORA-2024-633dc7e183 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2024-633dc7e183

FEDORA-2024-633dc7e183 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-633dc7e183`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-633dc7e183

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2024-53d986312e has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-53d986312e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-53d986312e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2024-53d986312e has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2024-633dc7e183 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.