Description of problem: Fail to restore a guest which have hot-pluged vcpus and numatune Version-Release number of selected component (if applicable): v2.1.0-209-ge3229f6 How reproducible: 100% Steps to Reproduce: 1. prepare a numa machine: # numactl --har available: 4 nodes (0-3) node 0 cpus: 0 2 4 6 8 10 node 0 size: 10205 MB node 0 free: 7497 MB node 1 cpus: 12 14 16 18 20 22 node 1 size: 8192 MB node 1 free: 5806 MB node 2 cpus: 1 3 5 7 9 11 node 2 size: 6144 MB node 2 free: 5446 MB node 3 cpus: 13 15 17 19 21 23 node 3 size: 8175 MB node 3 free: 7734 MB node distances: node 0 1 2 3 0: 10 20 20 20 1: 20 10 20 20 2: 20 20 10 20 3: 20 20 20 10 # cat /proc/zoneinfo |grep DMA Node 0, zone DMA Node 0, zone DMA32 2. start guest and bind guest memory to different node with DMA: # virsh dumpxml r7 <domain type='kvm'> <name>r7</name> <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid> <maxMemory slots='16' unit='KiB'>15242882</maxMemory> <memory unit='KiB'>2228224</memory> <currentMemory unit='KiB'>2228224</currentMemory> <vcpu placement='static' current='1'>12</vcpu> ... <numatune> <memory mode='strict' nodeset='1'/> </numatune> # virsh start r7 Domain r7 started 3. hotplug vcpu: # virsh setvcpus r7 10 4. save guest: # virsh managedsave r7 Domain r7 state saved by libvirt 5. restore guest: # virsh start r7 error: Failed to start domain r7 error: internal error: qemu unexpectedly closed the monitor:...... ...... check guest log: kvm_init_vcpu failed: Cannot allocate memory 6. edit save file and bind guest to 0-1 numa node and restart guest: # virsh save-image-edit /var/lib/libvirt/qemu/save/r7.save State file /var/lib/libvirt/qemu/save/r7.save edited. # virsh start r7 Domain r7 started Actual results: Fail to restore a guest which have hot-pluged vcpus and numatune Expected results: Libvirt need bind memory to all node when hotplug vcpus since qemu need access DMA when create new vcpu Additional info:
I found seems just start a guest which have new cpu device can reproduce this issue: 1. # virsh dumpxml r7 <domain type='kvm'> <name>r7</name> <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid> <maxMemory slots='16' unit='KiB'>15242882</maxMemory> <memory unit='KiB'>2228224</memory> <currentMemory unit='KiB'>2228224</currentMemory> <vcpu placement='static' current='3'>12</vcpu> <vcpus> <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/> <vcpu id='1' enabled='yes' hotpluggable='yes' order='2'/> <vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/> <vcpu id='3' enabled='no' hotpluggable='yes'/> <vcpu id='4' enabled='no' hotpluggable='yes'/> <vcpu id='5' enabled='no' hotpluggable='yes'/> <vcpu id='6' enabled='no' hotpluggable='yes'/> <vcpu id='7' enabled='no' hotpluggable='yes'/> <vcpu id='8' enabled='no' hotpluggable='yes'/> <vcpu id='9' enabled='no' hotpluggable='yes'/> <vcpu id='10' enabled='no' hotpluggable='yes'/> <vcpu id='11' enabled='no' hotpluggable='yes'/> </vcpus> 2. start guest # virsh start r7 error: Failed to start domain r7 error: internal error: qemu unexpectedly closed the monitor: ..... check guest log: kvm_init_vcpu failed: Cannot allocate memory qemu monitor: 57170.108 > 0x7f766c018fe0 {"execute":"device_add","arguments":{"driver":"Opteron_G4-x86_64-cpu","id":"vcpu1","socket-id":0,"core-id":1,"thread-id":0},"id":"libvirt-8"}
Fixed upstream: commit 68115fe0abf833a5d1dfb3a00aa216c332e84acb Author: Peter Krempa <pkrempa> Date: Wed Sep 7 13:20:00 2016 +0200 qemu: process: Fix start with unpluggable vcpus with NUMA pinning Similarly to vcpu hotplug the emulator thread cgroup numa mapping needs to be relaxed while hot-adding vcpus so that the threads can allocate data in the DMA zone. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1370084 commit eb5dee353479629d8769de0cfe77c59711c70d79 Author: Peter Krempa <pkrempa> Date: Wed Sep 7 13:11:59 2016 +0200 qemu: cgroup: Extract temporary relaxing of cgroup setting for vcpu hotplug When hot-adding vcpus qemu needs to allocate some structures in the DMA zone which may be outside of the numa pinning. Extract the code doing this in a set of helpers so that it can be reused.