Bug 1370084
| Summary: | Fail to start a guest which have hot-pluggable vcpus and numatune on a numa machine | ||
|---|---|---|---|
| Product: | [Community] Virtualization Tools | Reporter: | Luyao Huang <lhuang> |
| Component: | libvirt | Assignee: | Peter Krempa <pkrempa> |
| Status: | CLOSED NEXTRELEASE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | unspecified | CC: | libvirt-maint, pkrempa, rbalakri |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-07 14:13:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I found seems just start a guest which have new cpu device can reproduce this issue:
1.
# virsh dumpxml r7
<domain type='kvm'>
<name>r7</name>
<uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid>
<maxMemory slots='16' unit='KiB'>15242882</maxMemory>
<memory unit='KiB'>2228224</memory>
<currentMemory unit='KiB'>2228224</currentMemory>
<vcpu placement='static' current='3'>12</vcpu>
<vcpus>
<vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
<vcpu id='1' enabled='yes' hotpluggable='yes' order='2'/>
<vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/>
<vcpu id='3' enabled='no' hotpluggable='yes'/>
<vcpu id='4' enabled='no' hotpluggable='yes'/>
<vcpu id='5' enabled='no' hotpluggable='yes'/>
<vcpu id='6' enabled='no' hotpluggable='yes'/>
<vcpu id='7' enabled='no' hotpluggable='yes'/>
<vcpu id='8' enabled='no' hotpluggable='yes'/>
<vcpu id='9' enabled='no' hotpluggable='yes'/>
<vcpu id='10' enabled='no' hotpluggable='yes'/>
<vcpu id='11' enabled='no' hotpluggable='yes'/>
</vcpus>
2. start guest
# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor:
.....
check guest log:
kvm_init_vcpu failed: Cannot allocate memory
qemu monitor:
57170.108 > 0x7f766c018fe0 {"execute":"device_add","arguments":{"driver":"Opteron_G4-x86_64-cpu","id":"vcpu1","socket-id":0,"core-id":1,"thread-id":0},"id":"libvirt-8"}
Fixed upstream:
commit 68115fe0abf833a5d1dfb3a00aa216c332e84acb
Author: Peter Krempa <pkrempa>
Date: Wed Sep 7 13:20:00 2016 +0200
qemu: process: Fix start with unpluggable vcpus with NUMA pinning
Similarly to vcpu hotplug the emulator thread cgroup numa mapping needs
to be relaxed while hot-adding vcpus so that the threads can allocate
data in the DMA zone.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1370084
commit eb5dee353479629d8769de0cfe77c59711c70d79
Author: Peter Krempa <pkrempa>
Date: Wed Sep 7 13:11:59 2016 +0200
qemu: cgroup: Extract temporary relaxing of cgroup setting for vcpu hotplug
When hot-adding vcpus qemu needs to allocate some structures in the DMA
zone which may be outside of the numa pinning. Extract the code doing
this in a set of helpers so that it can be reused.
|
Description of problem: Fail to restore a guest which have hot-pluged vcpus and numatune Version-Release number of selected component (if applicable): v2.1.0-209-ge3229f6 How reproducible: 100% Steps to Reproduce: 1. prepare a numa machine: # numactl --har available: 4 nodes (0-3) node 0 cpus: 0 2 4 6 8 10 node 0 size: 10205 MB node 0 free: 7497 MB node 1 cpus: 12 14 16 18 20 22 node 1 size: 8192 MB node 1 free: 5806 MB node 2 cpus: 1 3 5 7 9 11 node 2 size: 6144 MB node 2 free: 5446 MB node 3 cpus: 13 15 17 19 21 23 node 3 size: 8175 MB node 3 free: 7734 MB node distances: node 0 1 2 3 0: 10 20 20 20 1: 20 10 20 20 2: 20 20 10 20 3: 20 20 20 10 # cat /proc/zoneinfo |grep DMA Node 0, zone DMA Node 0, zone DMA32 2. start guest and bind guest memory to different node with DMA: # virsh dumpxml r7 <domain type='kvm'> <name>r7</name> <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid> <maxMemory slots='16' unit='KiB'>15242882</maxMemory> <memory unit='KiB'>2228224</memory> <currentMemory unit='KiB'>2228224</currentMemory> <vcpu placement='static' current='1'>12</vcpu> ... <numatune> <memory mode='strict' nodeset='1'/> </numatune> # virsh start r7 Domain r7 started 3. hotplug vcpu: # virsh setvcpus r7 10 4. save guest: # virsh managedsave r7 Domain r7 state saved by libvirt 5. restore guest: # virsh start r7 error: Failed to start domain r7 error: internal error: qemu unexpectedly closed the monitor:...... ...... check guest log: kvm_init_vcpu failed: Cannot allocate memory 6. edit save file and bind guest to 0-1 numa node and restart guest: # virsh save-image-edit /var/lib/libvirt/qemu/save/r7.save State file /var/lib/libvirt/qemu/save/r7.save edited. # virsh start r7 Domain r7 started Actual results: Fail to restore a guest which have hot-pluged vcpus and numatune Expected results: Libvirt need bind memory to all node when hotplug vcpus since qemu need access DMA when create new vcpu Additional info: