Ever since QEMU commit 7b8be49d36fc ("NUMA: Enable adding NUMA node implicitly") in 2017 we started to implicitly create a single NUMA node when memory hotplug was possible (DIMM "slots" defined on the QEMU cmdline). Before that, the user had to create a dummy NUMA node manually on the QEMU cmdline. In QEMU commit 195784a0cfad ("numa: Auto-enable NUMA when any memory devices are possible") in 2020 we extended that handling for slot-less memory devices (like virtio-mem, see bz2216212). In Libvirt, in order to use memory devices (DIMMs/NVDIMMs/virtio-mem/virtio-pmem), we still have to manually create at least one NUMA node. Without a NUMA node, Libvirt will bail out when specifying the maxMemory option to enable memory devices like: <memory unit='GiB'>4</memory> <maxMemory slots='1' unit='GiB'>8</maxMemory> "error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug" And the user has to manually define a single NUMA node: <vcpu placement='static'>16</vcpu> ... <cpu ...> ... <numa> <cell id='0' cpus='0-16' memory='4' unit='GiB'/> </numa> </cpu> To get it working. To improve usability, either Libvirt should automatically create that single NUMA node (memory and VCPUs are known) for the user, or it should rely on QEMU to do it automatically. Letting Libvirt do it is probably cleaner, because that won't require QEMU compat handling.
Patch posted on the list: https://listman.redhat.com/archives/libvir-list/2023-July/240709.html
Merged upstream as: commit f5d4f5c8ee44e9f1939070afcc5381bdd5545e50 (laine/master) Author: Michal Prívozník <mprivozn> AuthorDate: Thu Jul 13 10:10:38 2023 +0200 Commit: Michal Prívozník <mprivozn> CommitDate: Tue Jul 18 08:42:55 2023 +0200 qemu: Add NUMA node automatically for memory hotplug Up until v2.11.0-rc2~19^2~3 QEMU used to require at least one NUMA node to be configured when memory hotplug was enabled. After that commit, QEMU automatically adds a NUMA node if none was specified on the cmd line. Reflect this in domain XML, i.e. explicitly add a NUMA node into our domain definition if needed. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2216236 Signed-off-by: Michal Privoznik <mprivozn> Reviewed-by: Kristina Hanicova <khanicov> v9.5.0-40-gf5d4f5c8ee
Tested on upstream build libvirt v9.5.0-41-gaece25f665 Hi michal, I found one issue below, 1. Define a vm with below xml: <maxMemory slots="2" unit="KiB">15242880</maxMemory> <memory unit="KiB">2097152</memory> <currentMemory unit="KiB">2048000</currentMemory> ... <memory model="dimm"> <target> <size unit="KiB">3145728</size> <node>0</node> </target> </memory> virsh define vm1.xml Domain 'vm1' defined from vm1.xml 2. Dump the config of the auto generated guest numa node # virsh dumpxml vm1 | xmllint -xpath '//numa' - <numa> <cell id="0" cpus="0-1" memory="18446744073708503040" unit="KiB"/> </numa> The memory of auto generated guest numa node is too big.
(In reply to liang cong from comment #3) > Tested on upstream build libvirt v9.5.0-41-gaece25f665 > > Hi michal, I found one issue below, > 1. Define a vm with below xml: > <maxMemory slots="2" unit="KiB">15242880</maxMemory> > <memory unit="KiB">2097152</memory> > <currentMemory unit="KiB">2048000</currentMemory> > ... > <memory model="dimm"> > <target> > <size unit="KiB">3145728</size> > <node>0</node> > </target> > </memory> > > virsh define vm1.xml > Domain 'vm1' defined from vm1.xml Yeah, this is because the <memory/> is maller than <memory model"dimm"/>. Without my commit, this would report an error: error: XML error: Total size of memory devices exceeds the total memory size Let me see if I can fix it.
https://listman.redhat.com/archives/libvir-list/2023-July/240892.html
The fix was merged: commit baeefe03279dc4b39e19ae61153aad864f311254 Author: Michal Prívozník <mprivozn> AuthorDate: Fri Jul 21 12:57:39 2023 +0200 Commit: Michal Prívozník <mprivozn> CommitDate: Tue Jul 25 14:51:35 2023 +0200 qemu_domain: Partially validate memory amounts when auto-adding NUMA node When automatically adding a NUMA node (qemuDomainDefNumaAutoAdd()) the memory size of the node is computed as: total_memory - sum(memory devices) And we have a nice helper for that: virDomainDefGetMemoryInitial() so it looks logical to just call it. Except, this code runs in post parse callback, i.e. memory sizes were not validated and it may happen that the sum is greater than the total memory. This would be caught by virDomainDefPostParseMemory() but that runs only after driver specific callbacks (i.e. after qemuDomainDefNumaAutoAdd()) and because the domain config was changed and memory was increased to this huge number no error is caught. So let's do what virDomainDefGetMemoryInitial() would do, but with error checking. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2216236 Fixes: f5d4f5c8ee44e9f1939070afcc5381bdd5545e50 Signed-off-by: Michal Privoznik <mprivozn> Reviewed-by: Kristina Hanicova <khanicov> v9.5.0-105-gbaeefe0327
Pre-verified on upstream build libvirt v9.6.0-rc1-9-ge33054b72c Test steps: Scenario 1: 1.1 Define a vm with below xml: <maxMemory slots="2" unit="KiB">15242880</maxMemory> <memory unit="KiB">2097152</memory> <currentMemory unit="KiB">2048000</currentMemory> # virsh define vm1.xml Domain 'vm1' defined from vm1.xml 1.2 Dump the config of the auto generated guest numa node # virsh dumpxml vm1 | xmllint -xpath '//memory|//maxMemory|//numa|//currentMemory|//vcpu' - <maxMemory slots="2" unit="KiB">153339990</maxMemory> <memory unit="KiB">2097152</memory> <currentMemory unit="KiB">2048000</currentMemory> <vcpu placement="static" current="1">3</vcpu> <numa> <cell id="0" cpus="0-2" memory="1572864" unit="KiB"/> </numa> <memory model="dimm"> <target> <size unit="KiB">524288</size> <node>0</node> </target> <address type="dimm" slot="0"/> </memory> 1.3 Start the guest # virsh start vm1 Domain 'vm1' started Scenario2: 1.1 Define a vm with below xml: <maxMemory slots="2" unit="KiB">15242880</maxMemory> <memory unit="KiB">2097152</memory> <currentMemory unit="KiB">2048000</currentMemory> ... <memory model="dimm"> <target> <size unit="KiB">524288</size> <node>0</node> </target> </memory> # virsh define vm1.xml Domain 'vm1' defined from vm1.xml 2.2 Dump the config of the auto generated guest numa node # virsh dumpxml vm1 | xmllint -xpath '//memory|//maxMemory|//numa|//currentMemory|//vcpu' - <maxMemory slots="2" unit="KiB">153339990</maxMemory> <memory unit="KiB">2097152</memory> <currentMemory unit="KiB">2048000</currentMemory> <vcpu placement="static" current="1">3</vcpu> <numa> <cell id="0" cpus="0-2" memory="2097152" unit="KiB"/> </numa> 2.3 Start the guest # virsh start vm1 Domain 'vm1' started Also test other scenarios, such as: 1. slots with no memory device, dimm, nvdimm, virtio-mem, virtio-pmem, multi memory device 1.1 device memory bigger|equal than memory 1.2 memory unit MB 1.3 vcpu numbers 2. 0 losts with no memory device, dimm, nvdimm, virtio-mem, virtio-pmem, multi memory device 3. no maxmemory 4. no node, correct node, inexsitent node 5. with numa node