This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2216236 - [RHEL9] Libvirt requires at least one NUMA node with memory devices
Summary: [RHEL9] Libvirt requires at least one NUMA node with memory devices
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: 9.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: liang cong
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-06-20 15:06 UTC by David Hildenbrand
Modified: 2023-09-22 17:55 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-22 17:55:32 UTC
Type: Bug
Target Upstream Version: 9.6.0
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-7545 0 None Migrated None 2023-09-22 17:55:37 UTC
Red Hat Issue Tracker RHELPLAN-160305 0 None None None 2023-06-20 15:09:15 UTC

Description David Hildenbrand 2023-06-20 15:06:35 UTC
Ever since QEMU commit 7b8be49d36fc ("NUMA: Enable adding NUMA node implicitly") in 2017 we started to implicitly create a single NUMA node when memory hotplug was possible (DIMM "slots" defined on the QEMU cmdline). Before that, the user had to create a dummy NUMA node manually on the QEMU cmdline.

In QEMU commit 195784a0cfad ("numa: Auto-enable NUMA when any memory devices are possible") in 2020 we extended that handling for slot-less memory devices (like virtio-mem, see bz2216212).

In Libvirt, in order to use memory devices (DIMMs/NVDIMMs/virtio-mem/virtio-pmem), we still have to manually create at least one NUMA node.

Without a NUMA node, Libvirt will bail out when specifying the maxMemory option to enable memory devices like:

<memory unit='GiB'>4</memory>
<maxMemory slots='1' unit='GiB'>8</maxMemory>

"error: unsupported configuration: At least one numa node has to be 
configured when enabling memory hotplug"

And the user has to manually define a single NUMA node:

  <vcpu placement='static'>16</vcpu>
  ...
  <cpu ...>
  ...
    <numa>
      <cell id='0' cpus='0-16' memory='4' unit='GiB'/>
    </numa>
  </cpu>

To get it working.

To improve usability, either Libvirt should automatically create that single NUMA node (memory and VCPUs are known) for the user, or it should rely on QEMU to do it automatically.

Letting Libvirt do it is probably cleaner, because that won't require QEMU compat handling.

Comment 1 Michal Privoznik 2023-07-14 11:43:02 UTC
Patch posted on the list:

https://listman.redhat.com/archives/libvir-list/2023-July/240709.html

Comment 2 Michal Privoznik 2023-07-18 06:52:31 UTC
Merged upstream as:

commit f5d4f5c8ee44e9f1939070afcc5381bdd5545e50 (laine/master)
Author:     Michal Prívozník <mprivozn>
AuthorDate: Thu Jul 13 10:10:38 2023 +0200
Commit:     Michal Prívozník <mprivozn>
CommitDate: Tue Jul 18 08:42:55 2023 +0200

    qemu: Add NUMA node automatically for memory hotplug
    
    Up until v2.11.0-rc2~19^2~3 QEMU used to require at least one
    NUMA node to be configured when memory hotplug was enabled. After
    that commit, QEMU automatically adds a NUMA node if none was
    specified on the cmd line. Reflect this in domain XML, i.e.
    explicitly add a NUMA node into our domain definition if needed.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2216236
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Kristina Hanicova <khanicov>

v9.5.0-40-gf5d4f5c8ee

Comment 3 liang cong 2023-07-21 09:25:55 UTC
Tested on upstream build libvirt v9.5.0-41-gaece25f665

Hi michal, I found one issue below,
1. Define a vm with below xml:
<maxMemory slots="2" unit="KiB">15242880</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
...
<memory model="dimm">
      <target>
        <size unit="KiB">3145728</size>
        <node>0</node>
      </target>
</memory>

virsh define vm1.xml
Domain 'vm1' defined from vm1.xml

2. Dump the config of the auto generated guest numa node
# virsh dumpxml vm1 | xmllint -xpath '//numa' -
<numa>
      <cell id="0" cpus="0-1" memory="18446744073708503040" unit="KiB"/>
</numa>

The memory of auto generated guest numa node is too big.

Comment 4 Michal Privoznik 2023-07-21 10:44:21 UTC
(In reply to liang cong from comment #3)
> Tested on upstream build libvirt v9.5.0-41-gaece25f665
> 
> Hi michal, I found one issue below,
> 1. Define a vm with below xml:
> <maxMemory slots="2" unit="KiB">15242880</maxMemory>
> <memory unit="KiB">2097152</memory>
> <currentMemory unit="KiB">2048000</currentMemory>
> ...
> <memory model="dimm">
>       <target>
>         <size unit="KiB">3145728</size>
>         <node>0</node>
>       </target>
> </memory>
> 
> virsh define vm1.xml
> Domain 'vm1' defined from vm1.xml

Yeah, this is because the <memory/> is maller than <memory model"dimm"/>. Without my commit, this would report an error:

error: XML error: Total size of memory devices exceeds the total memory size

Let me see if I can fix it.

Comment 6 Michal Privoznik 2023-07-25 12:53:42 UTC
The fix was merged:

commit baeefe03279dc4b39e19ae61153aad864f311254
Author:     Michal Prívozník <mprivozn>
AuthorDate: Fri Jul 21 12:57:39 2023 +0200
Commit:     Michal Prívozník <mprivozn>
CommitDate: Tue Jul 25 14:51:35 2023 +0200

    qemu_domain: Partially validate memory amounts when auto-adding NUMA node
    
    When automatically adding a NUMA node (qemuDomainDefNumaAutoAdd()) the
    memory size of the node is computed as:
    
      total_memory - sum(memory devices)
    
    And we have a nice helper for that: virDomainDefGetMemoryInitial() so
    it looks logical to just call it. Except, this code runs in post parse
    callback, i.e. memory sizes were not validated and it may happen that
    the sum is greater than the total memory. This would be caught by
    virDomainDefPostParseMemory() but that runs only after driver specific
    callbacks (i.e. after qemuDomainDefNumaAutoAdd()) and because the
    domain config was changed and memory was increased to this huge
    number no error is caught.
    
    So let's do what virDomainDefGetMemoryInitial() would do, but
    with error checking.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2216236
    Fixes: f5d4f5c8ee44e9f1939070afcc5381bdd5545e50
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Kristina Hanicova <khanicov>

v9.5.0-105-gbaeefe0327

Comment 7 liang cong 2023-07-31 03:48:22 UTC
Pre-verified on upstream build libvirt v9.6.0-rc1-9-ge33054b72c 

Test steps:
Scenario 1:
1.1 Define a vm with below xml:
<maxMemory slots="2" unit="KiB">15242880</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>


# virsh define vm1.xml
Domain 'vm1' defined from vm1.xml

1.2 Dump the config of the auto generated guest numa node
# virsh dumpxml vm1 | xmllint -xpath '//memory|//maxMemory|//numa|//currentMemory|//vcpu' -
<maxMemory slots="2" unit="KiB">153339990</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
<vcpu placement="static" current="1">3</vcpu>
<numa>
      <cell id="0" cpus="0-2" memory="1572864" unit="KiB"/>
    </numa>
<memory model="dimm">
      <target>
        <size unit="KiB">524288</size>
        <node>0</node>
      </target>
      <address type="dimm" slot="0"/>
    </memory>

1.3 Start the guest 
# virsh start vm1
Domain 'vm1' started

Scenario2:
1.1 Define a vm with below xml:
<maxMemory slots="2" unit="KiB">15242880</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
...
<memory model="dimm">
      <target>
        <size unit="KiB">524288</size>
        <node>0</node>
      </target>
</memory>

# virsh define vm1.xml
Domain 'vm1' defined from vm1.xml

2.2 Dump the config of the auto generated guest numa node
# virsh dumpxml vm1 | xmllint -xpath '//memory|//maxMemory|//numa|//currentMemory|//vcpu' -
<maxMemory slots="2" unit="KiB">153339990</maxMemory>
<memory unit="KiB">2097152</memory>
<currentMemory unit="KiB">2048000</currentMemory>
<vcpu placement="static" current="1">3</vcpu>
<numa>
      <cell id="0" cpus="0-2" memory="2097152" unit="KiB"/>
    </numa>


2.3 Start the guest 
# virsh start vm1
Domain 'vm1' started


Also test other scenarios, such as:
1. slots with no memory device, dimm, nvdimm, virtio-mem, virtio-pmem, multi memory device
  1.1 device memory bigger|equal than memory
  1.2 memory unit MB
  1.3 vcpu numbers
2. 0 losts with no memory device, dimm, nvdimm, virtio-mem, virtio-pmem, multi memory device
3. no maxmemory
4. no node, correct node, inexsitent node
5. with numa node

Comment 8 RHEL Program Management 2023-09-22 17:54:54 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 9 RHEL Program Management 2023-09-22 17:55:32 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.


Note You need to log in before you can comment on or make changes to this bug.