We have an openQA test of PXE booting. This involves two VMs. The first sets itself up as a DHCP and PXE (tftp) server; the second runs an install that relies on booting via PXE from the first.

On ppc64le, this test currently fails (it passes on x86_64 and aarch64). When the client boots, it successfully retrieves the bootloader (grub) from the server, but grub doesn't behave as it should: it just shows its "Minimal BASH-like line editing is supported..." screen and a "grub>" prompt. The server places a grub.cfg alongside the bootloader which contains an entry to boot anaconda which should automatically load after five seconds, but it seems grub just does not load this config file, for some reason.

I can pinpoint the problem to grub2 because I can work around it by using an older grub2. The "server" VM is running Fedora 38. We set up the bootloader (on ppc64le, this is different on each arch) by doing this:

dnf -y install grub2-tools-extra
grub2-mknetdir --net-directory=/var/lib/tftpboot
printf 'set default=0\nset timeout=5\n\nmenuentry \"Install Fedora 64-bit\"  --class fedora --class gnu-linux --class gnu --class os {\n  linux fedora/vmlinuz ip=dhcp inst.ks=file:///ks.cfg\n  initrd fedora/initrd.img\n}' >> /var/lib/tftpboot/boot/grub2/grub.cfg

so usually, we use `grub2-mknetdir` from the current stable F38 grub2-tools-extra package to set up the bootloader (then write the config file with printf). But now I hacked up the test to do this instead of the first step:

dnf -y install koji
koji download-build --arch=ppc64le --arch=noarch grub2-2.06-45.fc36
dnf -y downgrade *.rpm
dnf -y install grub2-tools-extra-2.06-45.fc36.ppc64le.rpm

so we use an old grub2-tools-extra from F36 instead (I picked this build as it's the one the server test would have been using around the time the test last passed, in August 2022, according to the result history; I haven't had time to look at the ppc64le tests for a while, which is why I'm only just getting to this). The other two steps are the same. With this setup, things work - grub behaves as expected and the client boots to anaconda. The only change is the version of grub2-tools-extra (and the other grub2 packages that are installed) used on the server end.

Reproducible: Always

Steps to Reproduce:
1. On ppc64le, set up a dhcp/PXE server which serves grub2 as the bootloader, with a config file in the correct location relative to the bootloader
2. Try and boot a system via PXE using the server
Actual Results:  
Client system gets stuck at a grub prompt

Expected Results:  
Client system should behave according to the grub config file

I'll do some 'bisecting' to try and pinpoint the exact grub2 build that broke this, will report back soon.