Bug 2004662

Assigned to Klaus for initial triage per bz process and age of bug created or assigned to virt-maint without triage.

(In reply to Eduardo Habkost from comment #0)
> When testing SMBIOS 3.0 support in SeaBIOS, we found another limit in
> SeaBIOS code that can be hit with too large VMs.
> 
> Quoting https://bugzilla.redhat.com/show_bug.cgi?id=1904267#c70:
> 
> (In reply to Brian Payton from bug #1904267 comment #64)
> > Copying SMBIOS 3.0 from 0x00006cd9 to 0x000f5d00
> > WARNING - Unable to allocate resource at romfile_loader_allocate:87!
> 
> Maybe SeaBIOS has an internal memory allocation size limit similar to the
> edk2 limitations tracked at bug 1982176.
> 
> Note that the TSEG size setting affects only UEFI (AFAIK), so we probably
> need something different to make larger SMBIOS tables work with SeaBIOS.

Eduardo,

 looks like we need to do something about large VMs RFEs - would you recommend we prioritize edk2 or seabios and why?

My thoughts: Looks like we're favoring new workloads towards UEFI (i.e., confidential-computing compatibility), so perhaps we should prioritize edk2 and leave SeaBIOS alone in this scenario (unless someone else proposes patches, always welcome of course)?

Gerd, any thoughts?

 -Klaus

> > > WARNING - Unable to allocate resource at romfile_loader_allocate:87!

> and leave SeaBIOS alone in this scenario (unless someone else proposes
> patches, always welcome of course)?

Making BUILD_MAX_HIGHTABLE (src/config.h) larger would be my first try.
Its 256k by default.

(In reply to Klaus Heinrich Kiwi from comment #2)
>  looks like we need to do something about large VMs RFEs - would you
> recommend we prioritize edk2 or seabios and why?

There's no clear answer to me.  I was going to recommend we prioritize edk2, but maybe fixing SeaBIOS will be a lot simpler than fixing edk2.

I think my recommendation would be: let's address the low-hanging fruits in SeaBIOS first, but focus on edk2 long term.

(In reply to Gerd Hoffmann from comment #3)
> > > > WARNING - Unable to allocate resource at romfile_loader_allocate:87!
> 
> > and leave SeaBIOS alone in this scenario (unless someone else proposes
> > patches, always welcome of course)?
> 
> Making BUILD_MAX_HIGHTABLE (src/config.h) larger would be my first try.
> Its 256k by default.

Gerd,

perhaps a quick scratch-build could be provided back to Eduardo so he could test this?

 -Klaus

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=40693634
http://brew-task-repos.usersys.redhat.com/repos/scratch/ghoffman/seabios/1.14.0/7.el9_b.bz2004662.1/

Hello Gerd,

      Thank you for the fix.  I verified this problem is resolved in the seabios_1.el9_0.rebase16.17 we received.  With /usr/share/seabios/bios-256k.bin we are able to boot a 24TB 32 socket 1728 vcpu virtual instance.  Using the community seabios, the same instance does not boot and hangs and shows the following in the debugcon file:

handle_smp: apic_id=0x575
Found 1728 cpu(s) max supported 2038 cpu(s)
Copying SMBIOS 3.0 from 0x00006d00 to 0x000f5ce0
WARNING - Unable to allocate resource at romfile_loader_allocate:87!
WARNING - internal error detected at romfile_loader_add_checksum:152!
WARNING - internal error detected at romfile_loader_add_pointer:129!
WARNING - internal error detected at romfile_loader_add_pointer:129!

      Have a good day.

Regards,

Brian

Thanks, Brian for testing.
Setting needinfo for Gerd just to make sure that he sees your results.

https://git.kraxel.org/cgit/seabios/log/?h=memory-testing

This patch series adds support to grow seabios high memory heap dynamically.
(seabios_1.el9_0.rebase16.17 simply made the heap larger unconditionally).

If someone wants compile and test feel free to do so and report back here.
Test rpms for centos stream 9 will follow once the 1.16 rebase landed.

Needs additional patches on top of the rebase, so testonly would be wrong.

Rebase just landed in centos stream 9, so here are the test builds:
  https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=44784485
  http://brew-task-repos.usersys.redhat.com/repos/scratch/ghoffman/seabios/1.16.0/1.el9.kraxel202204220945/

> This patch series adds support to grow seabios high memory heap dynamically.
> (seabios_1.el9_0.rebase16.17 simply made the heap larger unconditionally).

Upstream patch discussion indicates simply making the heap larger
unconditionally is perfectly fine.  Unused heap space will be released
before booting the OS so that approach will not waste memory.

(In reply to Gerd Hoffmann from comment #14)
> Needs additional patches on top of the rebase, so testonly would be wrong.
> 
> Rebase just landed in centos stream 9, so here are the test builds:
>   https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=44784485
>  
> http://brew-task-repos.usersys.redhat.com/repos/scratch/ghoffman/seabios/1.
> 16.0/1.el9.kraxel202204220945/

Thanks for the update Gerd, are these additional patches also upstream or you are planning to post them soon?

upstream discussion in progress:
https://www.mail-archive.com/seabios@seabios.org/msg12891.html
https://www.mail-archive.com/seabios@seabios.org/msg12900.html

latest series for upstream:
https://www.mail-archive.com/seabios@seabios.org/msg12908.html

One more revision (loks like that'll be the final one).
https://www.mail-archive.com/seabios@seabios.org/msg12917.html

test build for this:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=44957932
http://brew-task-repos.usersys.redhat.com/repos/scratch/ghoffman/seabios/1.16.0/1.el9.bz2004662.20220429.0800/

https://gitlab.com/redhat/centos-stream/src/seabios/-/merge_requests/3

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Tested with qemu-kvm-7.0.0-1.el9 and 8 TB memory/224 vcpus, It wroks as expected. So set status to VERIFIED. If I was wrong, please correct me. Thanks.

Versions:
kernel-5.14.0-100.el9.x86_64
qemu-kvm-7.0.0-4.el9
edk2-ovmf-20220221gitb24306f15d-2.el9.noarch
seabios-bin-1.16.0-3.el9.noarch

boot a guest with seabios. (add smbios-entry-point-type=64 into qemu command lines)

/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem,kernel-irqchip=split,smbios-entry-point-type=64 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 8388608 \
    -object memory-backend-ram,size=8388608M,id=mem-machine_mem  \
    -smp 224,maxcpus=224,cores=112,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server',ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,kvm_pv_unhalt=on \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/avocado_ti4stbke/monitor-qmpmonitor1-20220412-052832-OLNnOWJ3,server=on,wait=off  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=qmp_id_catch_monitor,path=/tmp/avocado_ti4stbke/monitor-catch_monitor-20220412-052832-OLNnOWJ3,server=on,wait=off  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id5VPWWY \
    -chardev socket,id=chardev_serial0,path=/tmp/avocado_ti4stbke/serial-serial0-20220412-052832-OLNnOWJ3,server=on,wait=off \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20220412-052832-OLNnOWJ3,path=/tmp/avocado_ti4stbke/seabios-20220412-052832-OLNnOWJ3,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20220412-052832-OLNnOWJ3,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel910-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:83:04:f4:b5:1d,id=id9CnYs6,netdev=idNXvszc,bus=pcie-root-port-3,addr=0x0  \
    -chardev file,id=firmware,path=/tmp/debug.log \
    -device isa-debugcon,iobase=0x402,chardev=firmware \
    -netdev tap,id=idNXvszc,vhost=on  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \

The guest boot successfully.

# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.
Table at 0x7FFF8A20.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: SeaBIOS
	Version: 1.16.0-3.el9
	Release Date: 04/01/2014
	Address: 0xE8000
	Runtime Size: 96 kB
	ROM Size: 64 kB
	Characteristics:
		BIOS characteristics not supported
		Targeted content distribution is supported
	BIOS Revision: 0.0

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (seabios bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8051