Bug 1838633

Created attachment 1690645 [details]
OOM kernel error

I am seeing the same behaviour on native (not tagged) network as well.

Hello Lukas,

Could you please set debug=all to get more information on where this out of memory is happening?

Hello Javier,

I was on call with a customer who tried the same on their hardware with the same result. We tried with

set debug="http,efinet,net"

unfortunately there is not much logged. Attaching the screenshot from their hardware but it reads the same. When we tried with debug=all it was never ending and it was rolling for hour, then the error appears, "Press a key to continue" actually just waits few seconds and then it tries to boot the system. So we are not able to capture reliably anything for you. Please advice how to capture the debug output, maybe I could create a video and then seek in the recording. Or maybe if you can give me an option to prevent grub from booting when this error appears.

You can probably reproduce this in libvirt too, you just need to boot big enough live CD. You can use Fedora or in our case Discovery Image which is 300 MB sized RHEL7/CentOS7 created with livecd-creator: http://downloads.theforeman.org/discovery/nightly/fdi-image-latest.tar

Created attachment 1691896 [details]
Screenshot from MEN server (Intel Atom)

Created attachment 1691898 [details]
Screen from libvirt with debug=all

I'm changing this to Fedora 32 since is a regression in GRUB 2.04. It doesn't affect RHEL7 and RHEL8.

The memory allocation failure happens in the verifiers framework, because it reads the files to be verified as a single chunk when passing to the verifiers modules (i.e: the tpm module). So the issue happens when a initrd image is large (in Lukas' test the initrd size is 237 MiB) and GRUB tries to verify it.

By default GRUB request a quarter of the memory available system for its heap, which is enough for most cases but it is not when using large initrd images and the verifiers framework. One option could be to change GRUB's default to request a bigger size of the available memory for the heap.

I found the issue and is that now that the tpm module is built-in the EFI binary, GRUB is allocating two buffers to read the initrd image.

One buffer is allocated in the linux EFI loader and used as a bounce buffer because some machines aren't able to DMA above 4GB during EFI.
The other buffer is allocated by the verifiers framework as mentioned in Comment 8, to read the file and pass it as a single chunk to the
tpm module and other modules using the verifiers API.

Since the initrd image is quite big, there isn't enough memory in the heap to allocate two buffers of that size. But when using the verifiers
framework, the read operation is just a memory copy from the buffer that was used to read the file in the verifiers open handler, so there
is no need for a bounce buffer in the linux EFI loader anymore.

FEDORA-2020-193b04db8e has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-193b04db8e

FEDORA-2020-193b04db8e has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-193b04db8e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-193b04db8e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-193b04db8e has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.