Bug 790583
Summary: | [RFE] Support for memory hot-plug | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Bhavna Sarathy <bsarathy> |
Component: | libvirt | Assignee: | Peter Krempa <pkrempa> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 7.0 | CC: | amit.shah, cwei, dwu, dyuan, gsun, hhuang, jshortt, jsuchane, juzhang, knoel, lhuang, lmiksik, ltroan, mkenneth, mzhan, pkrempa, rbinkhor, rpacheco, shyu, smayhew, tao, tburke, virt-maint |
Target Milestone: | beta | Keywords: | FutureFeature, Triaged |
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | libvirt-1.2.14-1.el7 | Doc Type: | Enhancement |
Doc Text: |
Feature:
Libvirt support to use the qemu memory hotplug feature to do proper "hardware based" memory hotplug.
Reason:
Hotplug can be used to dynamically reallocate memory resources among virtual machines.
Result:
...
|
Story Points: | --- |
Clone Of: | 618903 | Environment: | |
Last Closed: | 2015-11-19 05:35:09 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 618903, 1186428, 1246527 | ||
Bug Blocks: | 580953, 607248, 822996, 1193097, 1205796, 1250957, 1252514 |
Comment 5
Osier Yang
2012-06-19 08:20:46 UTC
Update the bug summary to hot-plug only since qemu-kvm only support memory in-plug. Bug 618903 - [RFE] Support for memory hot-plug The following commits are possible prerequisite refactors and cleanups: commit 3b9a26a3252e8184fa8f0749941f691c43f23f1e Author: Peter Krempa <pkrempa> Date: Mon Sep 29 17:38:18 2014 +0200 conf: Move definition of virDomainParseMemory Shove it to the top of the file so that it can be reused earlier. commit 0b73366945372049463ca0f1b5991de559b56ed6 Author: Peter Krempa <pkrempa> Date: Mon Oct 6 17:28:46 2014 +0200 Implement empty post parse callbacks for all drivers To allow easy implementation of a callback check this patch adds empty post parse callbacks to drivers that were missing them. commit 6908f8cab30eb38810abb517dcdeb9b537d1a524 Author: Peter Krempa <pkrempa> Date: Tue Sep 23 13:25:25 2014 +0200 qemu: monitor: Add functions for object hot-add/remove To allow live modification of device backends in qemu libvirt needs to be able to hot-add/remove "objects". Add monitor backend functions to allow this. This function will be used for hot-add/remove of RNG backends, IOThreads, memory backing objects, etc. commit ddf52d01f852a61db9f74d2f3f9742832a03f1e2 Author: Peter Krempa <pkrempa> Date: Wed Oct 15 09:41:49 2014 +0200 util: json: Add option to skip adding a JSON object if it is NULL Add a new option specifier that will optionally add a JSON key=value pair containing a nested object if the added object isn't NULL. commit 81069ae3f53e0c1f09e0842969ba6774912a27c6 Author: Peter Krempa <pkrempa> Date: Tue Sep 23 11:48:08 2014 +0200 util: json: Improve handling and docs for adding JSON objects The JSON structure constructor has an option to add JSON arrays to the constructed object. The description is inaccurate as it can add any json object even a dict. Change the docs to cover this option and reject adding NULL objects. commit 881c46595e787ed232fce708a3480f5823bdce93 Author: Peter Krempa <pkrempa> Date: Mon Sep 22 16:35:02 2014 +0200 util: json: Split out code to create json value objects Our qemu monitor code has a converter from key-value pairs to a json value object. I want to re-use the code later and having it part of the monitor command generator is inflexible. Split it out into a separate helper. commit 3444fdefb11ec4b528db7a4228d69502ae9db8e2 Author: Peter Krempa <pkrempa> Date: Mon Sep 22 13:42:02 2014 +0200 qemu: hotplug: Use typecasted switch statement when plugging new devices commit 3d6c07f7f804be936df800eed749d01873da5a45 Author: Peter Krempa <pkrempa> Date: Mon Sep 29 18:43:34 2014 +0200 conf: Add compile time check that devices were checked for ABI stability As in the device info iterator add a switch that will force the compiler to check that new device types are added to the ABI stability checker. commit e7d67e7badb3d6dbd27ff077e745a873364f681d Author: Peter Krempa <pkrempa> Date: Tue Oct 7 14:26:32 2014 +0200 conf: shmem: Add ABI stability check Although the device will probably inhibit migration add checks to make sure that the configuration change gets caught. commit ac5979edc4857eab7fe5be434de99f5b84394d9b Author: Peter Krempa <pkrempa> Date: Tue Oct 7 17:21:45 2014 +0200 conf: Improve adding of new address types Use typecasted switch statement and note the type used to select the address type in a comment. Another batch of fixes: commit e9a1c4384c4f83a8e0d5d98c80369ecfd5b3f2e0 Author: Peter Krempa <pkrempa> Date: Wed Oct 15 12:24:02 2014 +0200 qemu: Convert qemuDomainUpdateDeviceConfig to typecasted enum commit fa3701a94c42d91e9c176161e196f92741bee1df Author: Peter Krempa <pkrempa> Date: Wed Oct 15 12:24:02 2014 +0200 qemu: Convert qemuDomainDetachDeviceConfig to typecasted enum commit 2536b1b95290036e63fdb6024163ada063169823 Author: Peter Krempa <pkrempa> Date: Wed Oct 15 12:24:02 2014 +0200 qemu: Convert qemuDomainAttachDeviceConfig to typecasted enum commit 714dff938cb3c7c401bed0362412ce08087c2125 Author: Peter Krempa <pkrempa> Date: Wed Oct 15 12:24:02 2014 +0200 qemu: Convert qemuDomainUpdateDeviceLive to typecasted enum commit 9bb21f4287e4a1cbc9deff1d3a3589f6cabce44c Author: Peter Krempa <pkrempa> Date: Wed Oct 15 12:24:02 2014 +0200 qemu: Convert qemuDomainDetachDeviceLive to typecasted enum The following commits finish the rather large effort to add memory hotplug support to libvirt: commit 3b289a81eaa49fd96b829cf47ea5c6d84e4f65c2 Author: Peter Krempa <pkrempa> Date: Wed Jan 21 17:45:54 2015 +0100 qemu: Implement memory device hotunplug Add code to hot-remove memory devices from qemu. Unfortunately QEMU doesn't support this right now, so this is just for completenes. commit 9b4654f6f12fbabd3a3d2def58ddb0034971f701 Author: Peter Krempa <pkrempa> Date: Mon Oct 13 00:28:58 2014 +0200 qemu: Implement memory device hotplug Add code to hot-add memory devices to running qemu instances. commit 96094fb28d83a71598ae20e701b48d526f54d407 Author: Peter Krempa <pkrempa> Date: Wed Jan 21 15:49:44 2015 +0100 qemu: conf: Add support for memory device cold(un)plug Add a few helpers that allow to operate with memory device definitions on the domain config and use them to implement memory device coldplug in the qemu driver. commit 8b54bffbab423608dcf596761675e7d206049902 Author: Peter Krempa <pkrempa> Date: Fri Jan 9 10:40:37 2015 +0100 qemu: add support for memory devices Add support to start qemu instance with 'pc-dimm' device. Thanks to the refactors we are able to reuse the existing function to determine the parameters. v1.2.13-271-g3b289a8 (Note that the list of commits is not exhaustive) Verify this bug with libvirt-1.2.17-2.el7.x86_64 and qemu-kvm-rhev-2.3.0-12.el7.x86_64: Hot-plug memory device: 1. # virsh dumpxml test3 ... <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>1024000</memory> <currentMemory unit='KiB'>1024000</currentMemory> ... <cpu> <numa> <cell id='0' cpus='0-1' memory='512000' unit='KiB'/> <cell id='1' cpus='2-3' memory='512000' unit='KiB'/> </numa> </cpu> ... 2. start guest and recheck the running xml and qemu command line: # virsh dumpxml test3 ... <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>1024000</memory> <currentMemory unit='KiB'>1024000</currentMemory> ... <cpu> <numa> <cell id='0' cpus='0-1' memory='512000' unit='KiB'/> <cell id='1' cpus='2-3' memory='512000' unit='KiB'/> </numa> </cpu> ... # ps aux|grep qemu ... -m size=1024000k,slots=16,maxmem=25600000k ... -numa node,nodeid=0,cpus=0-1,mem=500 -numa node,nodeid=1,cpus=2-3,mem=500 ... 3. prepare a memory device xml: # cat /root/memdevice.xml ... <memory model='dimm'> <target> <size unit='KiB'>512000</size> <node>0</node> </target> </memory> ... 4. # virsh attach-device test3 /root/memdevice.xml Device attached successfully 5. # virsh dumpxml test3 ... <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>1536000</memory> <currentMemory unit='KiB'>1536000</currentMemory> ... <memory model='dimm'> <target> <size unit='KiB'>512000</size> <node>0</node> </target> <alias name='dimm0'/> <address type='dimm' slot='0' base='0x100000000'/> </memory> ... Test memory device bind to host node and with hugepages: a memory eater i have used: #include <stdio.h> #include <stdlib.h> int main() { void *a = NULL; while(1){ a = malloc(1024*4); if (a == NULL) break; } while(1){ } return 0; } 1. # virsh dumpxml test3 ... <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2621440</memory> <currentMemory unit='KiB'>2621440</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> </hugepages> </memoryBacking> ... <cpu mode='host-model'> <model fallback='forbid'/> <numa> <cell id='0' cpus='0-1' memory='1048576' unit='KiB'/> <cell id='1' cpus='2-3' memory='1048576' unit='KiB'/> </numa> </cpu> ... <memory model='dimm'> <source> <nodemask>1</nodemask> </source> <target> <size unit='KiB'>524288</size> <node>0</node> </target> </memory> 2. # virsh start test3 Domain test3 started # numastat -p `pidof qemu-kvm` Per-node process memory usage (in MBs) for PID 13918 (qemu-kvm) Node 0 Node 1 Total --------------- --------------- --------------- Huge 1024.00 0.00 1024.00 Heap 12.05 0.00 12.05 Stack 10.13 0.00 10.13 Private 98.77 9.38 108.14 ---------------- --------------- --------------- --------------- Total 1144.95 9.38 1154.32 3. run memeater in guest and check the numa struct in guest : IN GUEST: # numactl --hard available: 2 nodes (0-1) node 0 cpus: 0 1 4 5 node 0 size: 1535 MB node 0 free: 1150 MB node 1 cpus: 2 3 node 1 size: 1023 MB node 1 free: 923 MB node distances: node 0 1 0: 10 20 1: 20 10 #./memeater 4. recheck the numa status in host: # numastat -p `pidof qemu-kvm` Per-node process memory usage (in MBs) for PID 13918 (qemu-kvm) Node 0 Node 1 Total --------------- --------------- --------------- Huge 1024.00 0.00 1024.00 Heap 13.30 0.00 13.30 Stack 8.25 0.00 8.25 Private 1040.03 507.39 1547.42 ---------------- --------------- --------------- --------------- Total 2085.57 507.39 2592.96 5. attach a memory device which use node 0 and have a hugepage: # cat memdev2.xml <memory model='dimm'> <source> <pagesize unit='KiB'>1048576</pagesize> <nodemask>0</nodemask> </source> <target> <size unit='m'>1024</size> <node>0</node> </target> </memory> # virsh attach-device test3 memdev2.xml Device attached successfully 6. # numastat -p `pidof qemu-kvm` Per-node process memory usage (in MBs) for PID 13918 (qemu-kvm) Node 0 Node 1 Total --------------- --------------- --------------- Huge 2048.00 0.00 2048.00 Heap 203.01 0.00 203.01 Stack 8.20 0.00 8.20 Private 1040.73 507.40 1548.14 ---------------- --------------- --------------- --------------- Total 3299.95 507.40 3807.35 7. check numa struct in guest: IN GUEST: # numactl --hard available: 2 nodes (0-1) node 0 cpus: 0 1 4 5 node 0 size: 2559 MB node 0 free: 2268 MB node 1 cpus: 2 3 node 1 size: 1023 MB node 1 free: 918 MB node distances: node 0 1 0: 10 20 1: 20 10 Test cold-plug/cold-unplug: 1. # virsh dumpxml test3 --inactive <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2621440</memory> <currentMemory unit='KiB'>2621440</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> </hugepages> </memoryBacking> ... <cpu mode='host-model'> <model fallback='forbid'/> <numa> <cell id='0' cpus='0-1' memory='1048576' unit='KiB'/> <cell id='1' cpus='2-3' memory='1048576' unit='KiB'/> </numa> </cpu> ... <memory model='dimm'> <source> <nodemask>1</nodemask> <pagesize unit='KiB'>4</pagesize> </source> <target> <size unit='KiB'>524288</size> <node>0</node> </target> </memory> 2. # cat memdev.xml <memory model='dimm'> <target> <size unit='m'>500</size> <node>0</node> </target> </memory> # virsh attach-device test3 memdev.xml --config Device attached successfully 3. # virsh dumpxml test3 --inactive ... <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>3133440</memory> <currentMemory unit='KiB'>3133440</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> </hugepages> </memoryBacking> ... <memory model='dimm'> <source> <nodemask>1</nodemask> <pagesize unit='KiB'>4</pagesize> </source> <target> <size unit='KiB'>524288</size> <node>0</node> </target> </memory> <memory model='dimm'> <target> <size unit='KiB'>512000</size> <node>0</node> </target> </memory> 4. # virsh detach-device test3 memdev.xml --config Device detached successfully 5. <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>2621440</memory> <currentMemory unit='KiB'>2621440</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> </hugepages> </memoryBacking> <memory model='dimm'> <source> <nodemask>1</nodemask> <pagesize unit='KiB'>4</pagesize> </source> <target> <size unit='KiB'>524288</size> <node>0</node> </target> </memory> About hot-unplug: seems qemu have some change in the latest version: 1. # virsh detach-device rhel7.0-rhel memdevice.xml Device detached successfully 2. recheck xml and still found the memdevice, so that mean qemu still not return a device remove event. 3. and check in guest, memory still exist. But i noticed qemu will report a error in qemu-kvm-rhev-2.3.0-2.el7.x86_64: # virsh detach-device test3 memdev2.xml error: Failed to detach device from memdev2.xml error: internal error: unable to execute QEMU command 'device_del': acpi: device unplug request for not supported device type: pc-dimm And also test with qemu-kvm (qemu-kvm-1.5.3-97.el7.x86_64): 1. prepare a geust with maxmemory and start it: # virsh dumpxml r7 <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>1024000</memory> <currentMemory unit='KiB'>624000</currentMemory> # virsh start r7 error: Failed to start domain r7 error: unsupported configuration: memory hotplug isn't supported by this QEMU binary Hi peter, Would you please help to check if we can verify this bug with step comment 17 - 20 ? Thanks in advance for your reply. Retest memory hot-unplug with lastest qemu and kernel since qemu already support hot-unplug in bz 1120706: host: libvirt-1.2.17-7.el7.x86_64 qemu-kvm-rhev-2.3.0-21.el7.x86_64 guest: kernel-3.10.0-314.el7.x86_64 systemd-219-3.el7.x86_64 steps: 1. prepare a guest with Maxmemory and numa in XML(i use a guest memory nearly 4G to avoid the issue described in bz 1245892): # virsh dumpxml rhel7.0-rhel <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>4048896</memory> <currentMemory unit='KiB'>4048896</currentMemory> ... <numa> <cell id='0' cpus='0-1' memory='2024448' unit='KiB'/> <cell id='1' cpus='2-3' memory='2024448' unit='KiB'/> </numa> ... 2. start guest and change the /usr/lib/udev/rules.d/40-redhat.rules from ATTR{state}="online" to ATTR{sate}="online_movable" 3. prepare a 128M memory device xml like this: # cat memdevice.xml <memory model='dimm'> <source> <pagesize unit='KiB'>4</pagesize> <nodemask>0</nodemask> </source> <target> <size unit='m'>128</size> <node>0</node> </target> </memory> 4. hot-plug the memory device and check xml : # virsh attach-device rhel7.0-rhel memdevice.xml Device attached successfully # virsh dumpxml rhel7.0-rhel <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>4179968</memory> <currentMemory unit='KiB'>4179968</currentMemory> ... <numa> <cell id='0' cpus='0-1' memory='2024448' unit='KiB'/> <cell id='1' cpus='2-3' memory='2024448' unit='KiB'/> </numa> ... <memory model='dimm'> <source> <nodemask>0</nodemask> <pagesize unit='KiB'>4</pagesize> </source> <target> <size unit='KiB'>131072</size> <node>0</node> </target> <alias name='dimm0'/> <address type='dimm' slot='0' base='0x140000000'/> </memory> 5. hot-unplug memory device: # virsh detach-device rhel7.0-rhel memdevice.xml Device detached successfully # virsh dumpxml rhel7.0-rhel <maxMemory slots='16' unit='KiB'>25600000</maxMemory> <memory unit='KiB'>4048896</memory> <currentMemory unit='KiB'>4048896</currentMemory> ... <numa> <cell id='0' cpus='0-1' memory='2024448' unit='KiB'/> <cell id='1' cpus='2-3' memory='2024448' unit='KiB'/> </numa> ... And we can get the qemu event: # virsh qemu-monitor-event rhel7.0-rhel --loop event DEVICE_DELETED at 1441610033.702064 for domain rhel7.0-rhel: {"device":"dimm0","path":"/machine/peripheral/dimm0"} But i notice when the currentMemory != memory the result is not right, and i have add a comment in bug 1220702. The test results look okay to me. The issue pointed out in comment 22 should be discussed in the BZ that is corresponding to it. (In reply to Peter Krempa from comment #23) > The test results look okay to me. The issue pointed out in comment 22 should > be discussed in the BZ that is corresponding to it. Got it, thanks a lot for your reply and verify this bug. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2202.html |