Bug 2216236
| Summary: | [RHEL9] Libvirt requires at least one NUMA node with memory devices | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | David Hildenbrand <dhildenb> |
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> |
| libvirt sub component: | General | QA Contact: | liang cong <lcong> |
| Status: | POST --- | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | jsuchane, lmen, mprivozn, virt-maint |
| Version: | 9.2 | Keywords: | Triaged, Upstream |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | 9.6.0 |
| Embargoed: | |||
Patch posted on the list: https://listman.redhat.com/archives/libvir-list/2023-July/240709.html Merged upstream as:
commit f5d4f5c8ee44e9f1939070afcc5381bdd5545e50 (laine/master)
Author: Michal Prívozník <mprivozn>
AuthorDate: Thu Jul 13 10:10:38 2023 +0200
Commit: Michal Prívozník <mprivozn>
CommitDate: Tue Jul 18 08:42:55 2023 +0200
qemu: Add NUMA node automatically for memory hotplug
Up until v2.11.0-rc2~19^2~3 QEMU used to require at least one
NUMA node to be configured when memory hotplug was enabled. After
that commit, QEMU automatically adds a NUMA node if none was
specified on the cmd line. Reflect this in domain XML, i.e.
explicitly add a NUMA node into our domain definition if needed.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2216236
Signed-off-by: Michal Privoznik <mprivozn>
Reviewed-by: Kristina Hanicova <khanicov>
v9.5.0-40-gf5d4f5c8ee
Tested on upstream build libvirt v9.5.0-41-gaece25f665
Hi michal, I found one issue below,
1. Define a vm with below xml:
<maxMemory slots="2" unit="KiB">15242880</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
...
<memory model="dimm">
<target>
<size unit="KiB">3145728</size>
<node>0</node>
</target>
</memory>
virsh define vm1.xml
Domain 'vm1' defined from vm1.xml
2. Dump the config of the auto generated guest numa node
# virsh dumpxml vm1 | xmllint -xpath '//numa' -
<numa>
<cell id="0" cpus="0-1" memory="18446744073708503040" unit="KiB"/>
</numa>
The memory of auto generated guest numa node is too big.
(In reply to liang cong from comment #3) > Tested on upstream build libvirt v9.5.0-41-gaece25f665 > > Hi michal, I found one issue below, > 1. Define a vm with below xml: > <maxMemory slots="2" unit="KiB">15242880</maxMemory> > <memory unit="KiB">2097152</memory> > <currentMemory unit="KiB">2048000</currentMemory> > ... > <memory model="dimm"> > <target> > <size unit="KiB">3145728</size> > <node>0</node> > </target> > </memory> > > virsh define vm1.xml > Domain 'vm1' defined from vm1.xml Yeah, this is because the <memory/> is maller than <memory model"dimm"/>. Without my commit, this would report an error: error: XML error: Total size of memory devices exceeds the total memory size Let me see if I can fix it. The fix was merged:
commit baeefe03279dc4b39e19ae61153aad864f311254
Author: Michal Prívozník <mprivozn>
AuthorDate: Fri Jul 21 12:57:39 2023 +0200
Commit: Michal Prívozník <mprivozn>
CommitDate: Tue Jul 25 14:51:35 2023 +0200
qemu_domain: Partially validate memory amounts when auto-adding NUMA node
When automatically adding a NUMA node (qemuDomainDefNumaAutoAdd()) the
memory size of the node is computed as:
total_memory - sum(memory devices)
And we have a nice helper for that: virDomainDefGetMemoryInitial() so
it looks logical to just call it. Except, this code runs in post parse
callback, i.e. memory sizes were not validated and it may happen that
the sum is greater than the total memory. This would be caught by
virDomainDefPostParseMemory() but that runs only after driver specific
callbacks (i.e. after qemuDomainDefNumaAutoAdd()) and because the
domain config was changed and memory was increased to this huge
number no error is caught.
So let's do what virDomainDefGetMemoryInitial() would do, but
with error checking.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2216236
Fixes: f5d4f5c8ee44e9f1939070afcc5381bdd5545e50
Signed-off-by: Michal Privoznik <mprivozn>
Reviewed-by: Kristina Hanicova <khanicov>
v9.5.0-105-gbaeefe0327
Pre-verified on upstream build libvirt v9.6.0-rc1-9-ge33054b72c
Test steps:
Scenario 1:
1.1 Define a vm with below xml:
<maxMemory slots="2" unit="KiB">15242880</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
# virsh define vm1.xml
Domain 'vm1' defined from vm1.xml
1.2 Dump the config of the auto generated guest numa node
# virsh dumpxml vm1 | xmllint -xpath '//memory|//maxMemory|//numa|//currentMemory|//vcpu' -
<maxMemory slots="2" unit="KiB">153339990</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
<vcpu placement="static" current="1">3</vcpu>
<numa>
<cell id="0" cpus="0-2" memory="1572864" unit="KiB"/>
</numa>
<memory model="dimm">
<target>
<size unit="KiB">524288</size>
<node>0</node>
</target>
<address type="dimm" slot="0"/>
</memory>
1.3 Start the guest
# virsh start vm1
Domain 'vm1' started
Scenario2:
1.1 Define a vm with below xml:
<maxMemory slots="2" unit="KiB">15242880</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
...
<memory model="dimm">
<target>
<size unit="KiB">524288</size>
<node>0</node>
</target>
</memory>
# virsh define vm1.xml
Domain 'vm1' defined from vm1.xml
2.2 Dump the config of the auto generated guest numa node
# virsh dumpxml vm1 | xmllint -xpath '//memory|//maxMemory|//numa|//currentMemory|//vcpu' -
<maxMemory slots="2" unit="KiB">153339990</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
<vcpu placement="static" current="1">3</vcpu>
<numa>
<cell id="0" cpus="0-2" memory="2097152" unit="KiB"/>
</numa>
2.3 Start the guest
# virsh start vm1
Domain 'vm1' started
Also test other scenarios, such as:
1. slots with no memory device, dimm, nvdimm, virtio-mem, virtio-pmem, multi memory device
1.1 device memory bigger|equal than memory
1.2 memory unit MB
1.3 vcpu numbers
2. 0 losts with no memory device, dimm, nvdimm, virtio-mem, virtio-pmem, multi memory device
3. no maxmemory
4. no node, correct node, inexsitent node
5. with numa node
|
Ever since QEMU commit 7b8be49d36fc ("NUMA: Enable adding NUMA node implicitly") in 2017 we started to implicitly create a single NUMA node when memory hotplug was possible (DIMM "slots" defined on the QEMU cmdline). Before that, the user had to create a dummy NUMA node manually on the QEMU cmdline. In QEMU commit 195784a0cfad ("numa: Auto-enable NUMA when any memory devices are possible") in 2020 we extended that handling for slot-less memory devices (like virtio-mem, see bz2216212). In Libvirt, in order to use memory devices (DIMMs/NVDIMMs/virtio-mem/virtio-pmem), we still have to manually create at least one NUMA node. Without a NUMA node, Libvirt will bail out when specifying the maxMemory option to enable memory devices like: <memory unit='GiB'>4</memory> <maxMemory slots='1' unit='GiB'>8</maxMemory> "error: unsupported configuration: At least one numa node has to be configured when enabling memory hotplug" And the user has to manually define a single NUMA node: <vcpu placement='static'>16</vcpu> ... <cpu ...> ... <numa> <cell id='0' cpus='0-16' memory='4' unit='GiB'/> </numa> </cpu> To get it working. To improve usability, either Libvirt should automatically create that single NUMA node (memory and VCPUs are known) for the user, or it should rely on QEMU to do it automatically. Letting Libvirt do it is probably cleaner, because that won't require QEMU compat handling.