*** Bug 828574 has been marked as a duplicate of this bug. ***

Update the bug summary to hot-plug only since qemu-kvm only support memory in-plug.

Bug 618903 - [RFE] Support for memory hot-plug

The following commits are possible prerequisite refactors and cleanups:

commit 3b9a26a3252e8184fa8f0749941f691c43f23f1e
Author: Peter Krempa <pkrempa>
Date:   Mon Sep 29 17:38:18 2014 +0200

    conf: Move definition of virDomainParseMemory
    
    Shove it to the top of the file so that it can be reused earlier.

commit 0b73366945372049463ca0f1b5991de559b56ed6
Author: Peter Krempa <pkrempa>
Date:   Mon Oct 6 17:28:46 2014 +0200

    Implement empty post parse callbacks for all drivers
    
    To allow easy implementation of a callback check this patch adds empty
    post parse callbacks to drivers that were missing them.

commit 6908f8cab30eb38810abb517dcdeb9b537d1a524
Author: Peter Krempa <pkrempa>
Date:   Tue Sep 23 13:25:25 2014 +0200

    qemu: monitor: Add functions for object hot-add/remove
    
    To allow live modification of device backends in qemu libvirt needs to
    be able to hot-add/remove "objects". Add monitor backend functions to
    allow this.
    
    This function will be used for hot-add/remove of RNG backends,
    IOThreads, memory backing objects, etc.

commit ddf52d01f852a61db9f74d2f3f9742832a03f1e2
Author: Peter Krempa <pkrempa>
Date:   Wed Oct 15 09:41:49 2014 +0200

    util: json: Add option to skip adding a JSON object if it is NULL
    
    Add a new option specifier that will optionally add a JSON key=value pair
    containing a nested object if the added object isn't NULL.

commit 81069ae3f53e0c1f09e0842969ba6774912a27c6
Author: Peter Krempa <pkrempa>
Date:   Tue Sep 23 11:48:08 2014 +0200

    util: json: Improve handling and docs for adding JSON objects
    
    The JSON structure constructor has an option to add JSON arrays to the
    constructed object. The description is inaccurate as it can add any json
    object even a dict. Change the docs to cover this option and reject
    adding NULL objects.

commit 881c46595e787ed232fce708a3480f5823bdce93
Author: Peter Krempa <pkrempa>
Date:   Mon Sep 22 16:35:02 2014 +0200

    util: json: Split out code to create json value objects
    
    Our qemu monitor code has a converter from key-value pairs to a json
    value object. I want to re-use the code later and having it part of the
    monitor command generator is inflexible. Split it out into a separate
    helper.

commit 3444fdefb11ec4b528db7a4228d69502ae9db8e2
Author: Peter Krempa <pkrempa>
Date:   Mon Sep 22 13:42:02 2014 +0200

    qemu: hotplug: Use typecasted switch statement when plugging new devices

commit 3d6c07f7f804be936df800eed749d01873da5a45
Author: Peter Krempa <pkrempa>
Date:   Mon Sep 29 18:43:34 2014 +0200

    conf: Add compile time check that devices were checked for ABI stability
    
    As in the device info iterator add a switch that will force the compiler
    to check that new device types are added to the ABI stability checker.

commit e7d67e7badb3d6dbd27ff077e745a873364f681d
Author: Peter Krempa <pkrempa>
Date:   Tue Oct 7 14:26:32 2014 +0200

    conf: shmem: Add ABI stability check
    
    Although the device will probably inhibit migration add checks to make
    sure that the configuration change gets caught.

commit ac5979edc4857eab7fe5be434de99f5b84394d9b
Author: Peter Krempa <pkrempa>
Date:   Tue Oct 7 17:21:45 2014 +0200

    conf: Improve adding of new address types
    
    Use typecasted switch statement and note the type used to select the
    address type in a comment.

Another batch of fixes:

commit e9a1c4384c4f83a8e0d5d98c80369ecfd5b3f2e0
Author: Peter Krempa <pkrempa>
Date:   Wed Oct 15 12:24:02 2014 +0200

    qemu: Convert qemuDomainUpdateDeviceConfig to typecasted enum

commit fa3701a94c42d91e9c176161e196f92741bee1df
Author: Peter Krempa <pkrempa>
Date:   Wed Oct 15 12:24:02 2014 +0200

    qemu: Convert qemuDomainDetachDeviceConfig to typecasted enum

commit 2536b1b95290036e63fdb6024163ada063169823
Author: Peter Krempa <pkrempa>
Date:   Wed Oct 15 12:24:02 2014 +0200

    qemu: Convert qemuDomainAttachDeviceConfig to typecasted enum

commit 714dff938cb3c7c401bed0362412ce08087c2125
Author: Peter Krempa <pkrempa>
Date:   Wed Oct 15 12:24:02 2014 +0200

    qemu: Convert qemuDomainUpdateDeviceLive to typecasted enum

commit 9bb21f4287e4a1cbc9deff1d3a3589f6cabce44c
Author: Peter Krempa <pkrempa>
Date:   Wed Oct 15 12:24:02 2014 +0200

    qemu: Convert qemuDomainDetachDeviceLive to typecasted enum

The following commits finish the rather large effort to add memory hotplug support to libvirt:

commit 3b289a81eaa49fd96b829cf47ea5c6d84e4f65c2
Author: Peter Krempa <pkrempa>
Date:   Wed Jan 21 17:45:54 2015 +0100

    qemu: Implement memory device hotunplug
    
    Add code to hot-remove memory devices from qemu. Unfortunately QEMU
    doesn't support this right now, so this is just for completenes.

commit 9b4654f6f12fbabd3a3d2def58ddb0034971f701
Author: Peter Krempa <pkrempa>
Date:   Mon Oct 13 00:28:58 2014 +0200

    qemu: Implement memory device hotplug
    
    Add code to hot-add memory devices to running qemu instances.

commit 96094fb28d83a71598ae20e701b48d526f54d407
Author: Peter Krempa <pkrempa>
Date:   Wed Jan 21 15:49:44 2015 +0100

    qemu: conf: Add support for memory device cold(un)plug
    
    Add a few helpers that allow to operate with memory device definitions
    on the domain config and use them to implement memory device coldplug in
    the qemu driver.


commit 8b54bffbab423608dcf596761675e7d206049902
Author: Peter Krempa <pkrempa>
Date:   Fri Jan 9 10:40:37 2015 +0100

    qemu: add support for memory devices
    
    Add support to start qemu instance with 'pc-dimm' device. Thanks to the
    refactors we are able to reuse the existing function to determine the
    parameters.

v1.2.13-271-g3b289a8

(Note that the list of commits is not exhaustive)

Verify this bug with libvirt-1.2.17-2.el7.x86_64 and qemu-kvm-rhev-2.3.0-12.el7.x86_64:

Hot-plug memory device:

1.
# virsh dumpxml test3

...

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>1024000</currentMemory>

...

  <cpu>
    <numa>
      <cell id='0' cpus='0-1' memory='512000' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='512000' unit='KiB'/>
    </numa>
  </cpu>

...

 

2. start guest and recheck the running xml and qemu command line:

# virsh dumpxml test3

...

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>1024000</currentMemory>

...

  <cpu>
    <numa>
      <cell id='0' cpus='0-1' memory='512000' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='512000' unit='KiB'/>
    </numa>
  </cpu>
...

 

# ps aux|grep qemu

...

-m size=1024000k,slots=16,maxmem=25600000k
...

-numa node,nodeid=0,cpus=0-1,mem=500 -numa node,nodeid=1,cpus=2-3,mem=500

...

 

3.  prepare a memory device xml:

# cat /root/memdevice.xml

...

    <memory model='dimm'>
      <target>
        <size unit='KiB'>512000</size>
        <node>0</node>
      </target>
    </memory>
...

 

4.

# virsh attach-device test3 /root/memdevice.xml
Device attached successfully

 

5.

# virsh dumpxml test3

...

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>1536000</memory>
  <currentMemory unit='KiB'>1536000</currentMemory>
...

    <memory model='dimm'>
      <target>
        <size unit='KiB'>512000</size>
        <node>0</node>
      </target>
      <alias name='dimm0'/>
      <address type='dimm' slot='0' base='0x100000000'/>
    </memory>
...

Test memory device bind to host node and with hugepages:

a memory eater i have used:

#include <stdio.h>
#include <stdlib.h>

int main() {
    void *a = NULL;
    while(1){
        a = malloc(1024*4);
        if (a == NULL)
            break;
    }
    while(1){
    }

    return 0;
}


1.
# virsh dumpxml test3

...

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>2621440</memory>
  <currentMemory unit='KiB'>2621440</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>

...

  <cpu mode='host-model'>
    <model fallback='forbid'/>
    <numa>
      <cell id='0' cpus='0-1' memory='1048576' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>

...

    <memory model='dimm'>
      <source>
        <nodemask>1</nodemask>
      </source>
      <target>
        <size unit='KiB'>524288</size>
        <node>0</node> 
      </target>
    </memory>

2.
# virsh start test3
Domain test3 started


# numastat -p `pidof qemu-kvm`

Per-node process memory usage (in MBs) for PID 13918 (qemu-kvm)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                      1024.00            0.00         1024.00
Heap                        12.05            0.00           12.05
Stack                       10.13            0.00           10.13
Private                     98.77            9.38          108.14
----------------  --------------- --------------- ---------------
Total                     1144.95            9.38         1154.32

3.
 run memeater in guest and check the numa struct in guest :

IN GUEST:

 

# numactl --hard
available: 2 nodes (0-1)
node 0 cpus: 0 1 4 5
node 0 size: 1535 MB
node 0 free: 1150 MB
node 1 cpus: 2 3
node 1 size: 1023 MB
node 1 free: 923 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10 

#./memeater


4.
recheck the numa status in host:

# numastat -p `pidof qemu-kvm`

Per-node process memory usage (in MBs) for PID 13918 (qemu-kvm)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                      1024.00            0.00         1024.00
Heap                        13.30            0.00           13.30
Stack                        8.25            0.00            8.25
Private                   1040.03          507.39         1547.42
----------------  --------------- --------------- ---------------
Total                     2085.57          507.39         2592.96

5.
attach a memory device which use node 0 and have a hugepage:

# cat memdev2.xml
    <memory model='dimm'>
      <source>
        <pagesize unit='KiB'>1048576</pagesize>
        <nodemask>0</nodemask>
      </source>
      <target>
        <size unit='m'>1024</size>
        <node>0</node>
      </target>
    </memory>

# virsh attach-device test3 memdev2.xml
Device attached successfully

6.
# numastat -p `pidof qemu-kvm`

Per-node process memory usage (in MBs) for PID 13918 (qemu-kvm)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                      2048.00            0.00         2048.00
Heap                       203.01            0.00          203.01
Stack                        8.20            0.00            8.20
Private                   1040.73          507.40         1548.14
----------------  --------------- --------------- ---------------
Total                     3299.95          507.40         3807.35

7.
check numa struct in guest:

IN GUEST:

# numactl --hard
available: 2 nodes (0-1)
node 0 cpus: 0 1 4 5
node 0 size: 2559 MB
node 0 free: 2268 MB
node 1 cpus: 2 3
node 1 size: 1023 MB
node 1 free: 918 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

Test cold-plug/cold-unplug:

1.
# virsh dumpxml test3 --inactive

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>2621440</memory>
  <currentMemory unit='KiB'>2621440</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
...

  <cpu mode='host-model'>
    <model fallback='forbid'/>
    <numa>
      <cell id='0' cpus='0-1' memory='1048576' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>
...

    <memory model='dimm'>
      <source>
        <nodemask>1</nodemask>
        <pagesize unit='KiB'>4</pagesize>
      </source>
      <target>
        <size unit='KiB'>524288</size>
        <node>0</node>
      </target>
    </memory>

2.

# cat memdev.xml
    <memory model='dimm'>
      <target>
        <size unit='m'>500</size>
        <node>0</node>
      </target>
    </memory>

# virsh attach-device test3 memdev.xml  --config
Device attached successfully

3.

# virsh dumpxml test3 --inactive
...
  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>3133440</memory>
  <currentMemory unit='KiB'>3133440</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
...
    <memory model='dimm'>
      <source>
        <nodemask>1</nodemask>
        <pagesize unit='KiB'>4</pagesize>
      </source>
      <target>
        <size unit='KiB'>524288</size>
        <node>0</node>
      </target>
    </memory>
    <memory model='dimm'>
      <target>
        <size unit='KiB'>512000</size>
        <node>0</node>
      </target>
    </memory>

4.

# virsh detach-device test3 memdev.xml --config
Device detached successfully

5.

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>2621440</memory>
  <currentMemory unit='KiB'>2621440</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>

    <memory model='dimm'>
      <source>
        <nodemask>1</nodemask>
        <pagesize unit='KiB'>4</pagesize>
      </source>
      <target>
        <size unit='KiB'>524288</size>
        <node>0</node>
      </target>
    </memory>

About hot-unplug:

seems qemu have some change in the latest version:

1.
# virsh detach-device rhel7.0-rhel memdevice.xml
Device detached successfully

2. recheck xml and still found the memdevice, so that mean qemu still not return a device remove event.

3. and check in guest, memory still exist.

But i noticed qemu will report a error in qemu-kvm-rhev-2.3.0-2.el7.x86_64:

# virsh detach-device test3 memdev2.xml 
error: Failed to detach device from memdev2.xml
error: internal error: unable to execute QEMU command 'device_del': acpi: device unplug request for not supported device type: pc-dimm


And also test with qemu-kvm (qemu-kvm-1.5.3-97.el7.x86_64):

1. prepare a geust with maxmemory and start it:
# virsh dumpxml r7

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>624000</currentMemory>


# virsh start r7
error: Failed to start domain r7
error: unsupported configuration: memory hotplug isn't supported by this QEMU binary

Hi peter,

Would you please help to check if we can verify this bug with step comment 17 - 20 ?

Thanks in advance for your reply.

Retest memory hot-unplug with lastest qemu and kernel since qemu already support hot-unplug in bz 1120706:

host:
libvirt-1.2.17-7.el7.x86_64
qemu-kvm-rhev-2.3.0-21.el7.x86_64

guest:
kernel-3.10.0-314.el7.x86_64
systemd-219-3.el7.x86_64

steps:

1. prepare a guest with Maxmemory and numa in XML(i use a guest memory nearly 4G to avoid the issue described in bz 1245892):

# virsh dumpxml rhel7.0-rhel

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>4048896</memory>
  <currentMemory unit='KiB'>4048896</currentMemory>
...
    <numa>
      <cell id='0' cpus='0-1' memory='2024448' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='2024448' unit='KiB'/>
    </numa>
...

2. start guest and change the /usr/lib/udev/rules.d/40-redhat.rules
from ATTR{state}="online" to ATTR{sate}="online_movable"

3. prepare a 128M memory device xml like this:
# cat memdevice.xml 
    <memory model='dimm'>
      <source>
        <pagesize unit='KiB'>4</pagesize>
        <nodemask>0</nodemask>
      </source>
      <target>
        <size unit='m'>128</size>
        <node>0</node>
      </target>
    </memory>

4. hot-plug the memory device and check xml :

# virsh attach-device rhel7.0-rhel memdevice.xml
Device attached successfully

# virsh dumpxml rhel7.0-rhel

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>4179968</memory>
  <currentMemory unit='KiB'>4179968</currentMemory>
...
    <numa>
      <cell id='0' cpus='0-1' memory='2024448' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='2024448' unit='KiB'/>
    </numa>
...
    <memory model='dimm'>
      <source>
        <nodemask>0</nodemask>
        <pagesize unit='KiB'>4</pagesize>
      </source>
      <target>
        <size unit='KiB'>131072</size>
        <node>0</node>
      </target>
      <alias name='dimm0'/>
      <address type='dimm' slot='0' base='0x140000000'/>
    </memory>

5. hot-unplug memory device:

# virsh detach-device rhel7.0-rhel memdevice.xml
Device detached successfully

# virsh dumpxml rhel7.0-rhel

  <maxMemory slots='16' unit='KiB'>25600000</maxMemory>
  <memory unit='KiB'>4048896</memory>
  <currentMemory unit='KiB'>4048896</currentMemory>
...
    <numa>
      <cell id='0' cpus='0-1' memory='2024448' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='2024448' unit='KiB'/>
    </numa>
...

And we can get the qemu event:

# virsh qemu-monitor-event rhel7.0-rhel --loop
event DEVICE_DELETED at 1441610033.702064 for domain rhel7.0-rhel: {"device":"dimm0","path":"/machine/peripheral/dimm0"}


But i notice when the currentMemory != memory the result is not right, and i have add a comment in bug 1220702.

The test results look okay to me. The issue pointed out in comment 22 should be discussed in the BZ that is corresponding to it.

(In reply to Peter Krempa from comment #23)
> The test results look okay to me. The issue pointed out in comment 22 should
> be discussed in the BZ that is corresponding to it.

Got it, thanks a lot for your reply and verify this bug.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html