Hide Forgot
Description of problem: An error message found in host "xm dmesg" when try to start a HVM guest with 881G or larger memory: (XEN) HVM33: Writing SMBIOS tables ... (XEN) HVM33: Could not write SMBIOS tables, error in hvmloader.c:hvm_write_smbios_tables() (XEN) HVM33: Loading ROMBIOS ... (XEN) HVM33: 8876 bytes of ROMBIOS high-memory extensions: (XEN) HVM33: Relocating to 0xefffac00-0xefffceac ... done ... The dmidecode output is empty after login the guest. Version-Release number of selected component (if applicable): Host: kernel-xen-2.6.18-294.el5xen xen-3.0.3-135.el5 Guest (RHEL6.2): 2.6.32-214.el6.x86_64 Also with RHEL6.1, RHEL5.7 HVM guest. How reproducible: 100% Steps to Reproduce: 1. configure memory size as 881G (881*1024) in guest conf: 2. create the guest, then check the guest log in "xm dmesg", the error message will be found: "Could not write SMBIOS tables, error in hvmloader.c:hvm_write_smbios_tables()" 3. check "dmidecode" inside the guest, there is no any output No such problem found when starting guest with 880G or lower memory size. Actual results: Error message in "xm dmesg" and guest BIOS is empty. Expected results: No such message and guest BIOS is not empty, or write down the memory limitation into Tech Paper. Additional info: Host CPU Model: Intel Xeon E7450 Host Product Name: IBM 3850 M2 / x3950 M2 -[72335SC]- # xm info | grep mem total_memory : 1031166 free_memory : 1010119
Created attachment 532184 [details] xmdmesg-smbios.txt
When the SMBIOS tables are built in write_smbios_tables() we create a table for every 16GB, which would be 56 tables. Then apparently the end result (size of all the SMBIOS tables together) exceeds a page (4096 bytes). It's necessary for them to fit within a single page because they're only given this one page, i.e. the tables must be between SMBIOS_PHYSICAL_ADDRESS (E9000) and ACPI_PHYSICAL_ADDRESS (EA000). It's not clear how to fix this to me.
Upstream has changesets 23458 and 23460, and something before those, need to look at hg blame.
23203, 23206. Before those 18977 seems to have relocated the tables, and quintupled SMBIOS_MAXIMUM_SIZE to 0x5000.
... plus moved the SMBIOS area above the ACPI area, to 0xEB000..0xEFFFF. I think we should try backporting these two macro changes (SMBIOS_MAXIMUM_SIZE and SMBIOS_PHYSICAL_ADDRESS). I think hvm_write_smbios_tables() first calls write_smbios_tables() to prepare the tables (without checking for any size limit) from 0xC0000, then checking the size and if small enough, copying from 0xC0000 to SMBIOS_PHYSICAL_ADDRESS (now at 0xEB000.) It seems to trample over the VGA BIOS in any case (from 0xC0000) -- I guess that might be prepared later on, so we're free to use its final area temporarily? Anyway, 0xC0000 to 0xC4FFF (inclusive) doesn't seem to clash with the next area (etherboot, 0xC8000).
Created attachment 532251 [details] [1/1] move SMBIOS area above ACPI area VGABIOS area when used as scratch space for SMBIOS area, should accomodate new SMBIOS size. --- tools/firmware/hvmloader/config.h | 4 ++-- tools/firmware/hvmloader/hvmloader.c | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-)
> HVMLoader assertion '(ACPI_PHYSICAL_ADDRESS + acpi_sz) <= > SMBIOS_PHYSICAL_ADDRESS' failed at hvmloader.c:404 Apparently 0x1000 bytes are not enough for the ACPI tables. #define ACPI_PHYSICAL_ADDRESS 0x000EA000 #define SMBIOS_PHYSICAL_ADDRESS 0x000EB000 #define SMBIOS_MAXIMUM_SIZE 0x00005000 #define ROMBIOS_PHYSICAL_ADDRESS 0x000F0000 I think we should keep the new layout, and increase SMBIOS_PHYSICAL_ADDRESS and decrease SMBIOS_MAXIMUM_SIZE, so that (a) their sum remains equal to ROMBIOS_PHYSICAL_ADDRESS, and (b) SMBIOS_MAXIMUM_SIZE is big enough for this machine, and (c) SMBIOS_PHYSICAL_ADDRESS - ACPI_PHYSICAL_ADDRESS is big enough for the ACPI tables. How much guest memory do we want to support? (1) For example, setting SMBIOS_MAXIMUM_SIZE to 0x2000 from the original 0x1000, instead of 0x5000, should still more than double the supported RAM size, and probably (hopefully) leave enough space for ACPI. (2) If, however, acpi_build_tables() requires a fixed (guest-independent) number of bytes, then we could work upwards from ACPI_PHYSICAL_ADDRESS. Paolo, can we put a general limit on "acpi_sz"? acpi_build_tables() seems to be the root of quite a call tree, so if you have a negative answer immediately (ie. the cumulative size of the ACPI tables is guest or host dependent), then it would save me tracking down the "worst case" of acpi_build_tables(), and allow us to start experimenting with (1). Thanks.
(In reply to comment #9) > How much guest memory do we want to support? > We claim that we support 512G, and I'd guess that's the most we'll ever claim, so we probably don't need to fix this at all. That said, if[*] your (1) below is safe to do, then I don't see why we wouldn't want to try for 1T (which is the max the Xen HV supports, I believe). > (1) For example, setting SMBIOS_MAXIMUM_SIZE to 0x2000 from the original > 0x1000, instead of 0x5000, should still more than double the supported RAM > size, and probably (hopefully) leave enough space for ACPI. * The reason I said it wasn't clear to me how to fix this issue is because moving sections around seemed risky to me. I'm probably just overly paranoid though, so I'll happily take other opinions on the matter.
(In reply to comment #10) > (In reply to comment #9) > > How much guest memory do we want to support? > > > > We claim that we support 512G, and I'd guess that's the most we'll ever claim, > so we probably don't need to fix this at all. That said, if[*] your (1) below > is safe to do, then I don't see why we wouldn't want to try for 1T (which is > the max the Xen HV supports, I believe). I tried to look at the RHEL virtualization support matrix and the RHEL-5 Virt guide: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Virtualization_limitations-Xen_limitations.html It only speaks about 32 bit HVM guests when it comes to memory. Had I known we explicitly state a limit of 512 GB, I wouldn't even have bothered checking the source. Qin wrote in comment 0: Expected results: No such message and guest BIOS is not empty, or write down the memory limitation into Tech Paper. So if the limit is already documented somewhere, then I believe we should immediately close it as NOTABUG. > > (1) For example, setting SMBIOS_MAXIMUM_SIZE to 0x2000 from the original > > 0x1000, instead of 0x5000, should still more than double the supported RAM > > size, and probably (hopefully) leave enough space for ACPI. > > * The reason I said it wasn't clear to me how to fix this issue is because > moving sections around seemed risky to me. I'm probably just overly paranoid > though, so I'll happily take other opinions on the matter. I figured if I break something with the layout then the boot will immediately blow up. Plus seeing upstream touch that code was permission to muck around. But, again, at that time I didn't know about the express 512 GB limit. I vote NOTABUG (or WONTFIX anyway).
512 GB ought to be enough for everyone.