Bug 857312
| Summary: | Failed to run numatune with specified node | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | hongming <honzhang> | |
| Component: | libvirt | Assignee: | Ján Tomko <jtomko> | |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 6.4 | CC: | cwei, dyuan, gsun, jiahu, jmiao, jsvarova, lsu, mtessun, mzhan, rbalakri, tdosek | |
| Target Milestone: | rc | Keywords: | Upstream, ZStream | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | libvirt-0.10.2-40.el6 | Doc Type: | Bug Fix | |
| Doc Text: |
Prior to this update, the libvirt library was trying to pin the domain's parent thread to another CPU even if its vCPU threads were not allowed to run on that CPU. As a consequence, the pinning process failed. To fix this bug, the child threads repin to the old and new CPUs first before only pinning them to the new ones. As a result, pinning now works even if the new CPU list is not a subset of the old CPUs.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 920406 1112820 (view as bug list) | Environment: | ||
| Last Closed: | 2014-10-14 04:14:16 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 920406, 994246, 1056252, 1112820 | |||
|
Description
hongming
2012-09-14 05:30:35 UTC
The issue can be reproduced on normal machine not NUMA, but got different
error message("Failed to write value '1': Invalid argument") in libvirtd.log.
Anyway, the "Unknown error" is not clear for users.
# virsh nodeinfo
CPU model: x86_64
CPU(s): 2
CPU frequency: 2992 MHz
CPU socket(s): 1
Core(s) per socket: 2
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 7864132 KiB
# virsh start foo
Domain foo started
# virsh numatune foo
numa_mode : strict
numa_nodeset : 0
# virsh numatune foo --nodeset 1
error: Unable to change numa parameters
error: unable to set numa tunable: Unknown error 18446744073709551615
# tailf /var/log/libvirt/libvirtd.log
<snip>
2012-09-14 03:44:54.602+0000: 4580: debug : virDomainSetNumaParameters:3943 : dom=0x7fe530002c40, (VM: name=foo, uuid=cae86633-904f-1c90-6bc4-9c579f70e699), params=0x7fe530003320, nparams=1, flags=0
2012-09-14 03:44:54.602+0000: 4580: debug : virCgroupNew:617 : New group /libvirt/qemu/foo
2012-09-14 03:44:54.602+0000: 4580: debug : virCgroupDetect:273 : Detected mount/mapping 0:cpu at /cgroup/cpu in
2012-09-14 03:44:54.602+0000: 4580: debug : virCgroupDetect:273 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in
2012-09-14 03:44:54.602+0000: 4580: debug : virCgroupDetect:273 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in
2012-09-14 03:44:54.602+0000: 4580: debug : virCgroupDetect:273 : Detected mount/mapping 3:memory at /cgroup/memory in
2012-09-14 03:44:54.602+0000: 4580: debug : virCgroupDetect:273 : Detected mount/mapping 4:devices at /cgroup/devices in
2012-09-14 03:44:54.602+0000: 4580: debug : virCgroupDetect:273 : Detected mount/mapping 5:freezer at /cgroup/freezer in
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupDetect:273 : Detected mount/mapping 6:blkio at /cgroup/blkio in
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:535 : Make group /libvirt/qemu/foo
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:560 : Make controller /cgroup/cpu/libvirt/qemu/foo/
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:560 : Make controller /cgroup/cpuacct/libvirt/qemu/foo/
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:560 : Make controller /cgroup/cpuset/libvirt/qemu/foo/
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:560 : Make controller /cgroup/memory/libvirt/qemu/foo/
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:560 : Make controller /cgroup/devices/libvirt/qemu/foo/
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:560 : Make controller /cgroup/freezer/libvirt/qemu/foo/
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupMakeGroup:560 : Make controller /cgroup/blkio/libvirt/qemu/foo/
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupSetValueStr:331 : Set value '/cgroup/cpuset/libvirt/qemu/foo/cpuset.mems' to '1'
2012-09-14 03:44:54.603+0000: 4580: debug : virCgroupSetValueStr:335 : Failed to write value '1': Invalid argument
2012-09-14 03:44:54.603+0000: 4580: error : qemuDomainSetNumaParameters:7549 : unable to set numa tunable: Unknown error 18446744073709551615
2012-09-14 03:44:54.603+0000: 4580: debug : virDomainFree:2278 : dom=0x7fe530002c40, (VM: name=foo, uuid=cae86633-904f-1c90-6bc4-9c579f70e699)
2012-09-14 03:44:54.604+0000: 4578: debug : virConnectClose:1446 : conn=0x7fe534345a70
</snip>
# rpm -q libvirt qemu-kvm-rhev kernel
libvirt-0.10.2-6.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.317.el6.x86_64
kernel-2.6.32-330.el6.x86_64
reproduce steps:
1. prepare a running domain
# virsh list
Id Name State
----------------------------------------------------
19 libvirt_test_api running
2. numatune with --live or --current
# virsh numatune libvirt_test_api strict --nodeset 1 --live
error: Unable to change numa parameters
error: unable to set numa tunable: Unknown error 18446744073709551615
# virsh numatune libvirt_test_api strict --nodeset 1 --current
error: Unable to change numa parameters
error: unable to set numa tunable: Unknown error 18446744073709551615
3. numatune with --config
# virsh numatune libvirt_test_api strict --nodeset 0 --config
# virsh destroy libvirt_test_api
Domain libvirt_test_api destroyed
# virsh dumpxml libvirt_test_api
...
<vcpu placement='static'>1</vcpu>
<numatune>
<memory mode='strict' nodeset='0'/>
</numatune>
...
# virsh start libvirt_test_api
Domain libvirt_test_api started
numatune with live or current will fail
(In reply to comment #0) > Description of problem: > Failed to run numatune with specified node. The following errors occur. > error: Unable to change numa parameters > error: unable to set numa tunable: Unknown error 18446744073709551615 > > Version-Release number of selected component (if applicable): > libvirt-0.10.1-2.el6.x86_64 > qemu-kvm-0.12.1.2-2.307.el6.x86_64 > > > How reproducible: > 100% > > Steps to Reproduce: > 1.# virsh nodeinfo > CPU model: x86_64 > CPU(s): 160 > CPU frequency: 1064 MHz > CPU socket(s): 1 > Core(s) per socket: 10 > Thread(s) per core: 2 > NUMA cell(s): 8 > Memory size: 1058739720 KiB > > 2. # virsh numatune testnuma1 > numa_mode : strict > numa_nodeset : 0 > > > 3. # virsh numatune testnuma1 --nodeset 1 > error: Unable to change numa parameters > error: unable to set numa tunable: Unknown error 18446744073709551615 Can you try to see what the result is with this: echo 1 > /cgroup/cpuset/libvirt/qemu/testnuma1/cpuset.mems > > 4.# vim /var/log/libvirt/libvirtd.log > > 2012-09-14 03:20:29.353+0000: 29132: debug : virCgroupNew:617 : New group > /libvirt/qemu/testnuma1 > 2012-09-14 03:20:29.354+0000: 29132: debug : virCgroupDetect:273 : Detected > mount/mapping 0:cpu at /cgroup/cpu in > 2012-09-14 03:20:29.354+0000: 29132: debug : virCgroupDetect:273 : Detected > mount/mapping 1:cpuacct at /cgroup/cpuacct in > 2012-09-14 03:20:29.354+0000: 29132: debug : virCgroupDetect:273 : Detected > mount/mapping 2:cpuset at /cgroup/cpuset in > 2012-09-14 03:20:29.354+0000: 29132: debug : virCgroupDetect:273 : Detected > mount/mapping 3:memory at /cgroup/memory in > 2012-09-14 03:20:29.354+0000: 29132: debug : virCgroupDetect:273 : Detected > mount/mapping 4:devices at /cgroup/devices in > 2012-09-14 03:20:29.354+0000: 29132: debug : virCgroupDetect:273 : Detected > mount/mapping 5:freezer at /cgroup/freezer in > 2012-09-14 03:20:29.354+0000: 29132: debug : virCgroupDetect:273 : Detected > mount/mapping 6:blkio at /cgroup/blkio in > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:535 : Make > group /libvirt/qemu/testnuma1 > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:560 : Make > controller /cgroup/cpu/libvirt/qemu/testnuma1/ > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:560 : Make > controller /cgroup/cpuacct/libvirt/qemu/testnuma1/ > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:560 : Make > controller /cgroup/cpuset/libvirt/qemu/testnuma1/ > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:560 : Make > controller /cgroup/memory/libvirt/qemu/testnuma1/ > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:560 : Make > controller /cgroup/devices/libvirt/qemu/testnuma1/ > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:560 : Make > controller /cgroup/freezer/libvirt/qemu/testnuma1/ > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupMakeGroup:560 : Make > controller /cgroup/blkio/libvirt/qemu/testnuma1/ > 2012-09-14 03:20:29.355+0000: 29132: debug : virCgroupSetValueStr:331 : Set > value '/cgroup/cpuset/libvirt/qemu/testnuma1/cpuset.mems' to '1' > 2012-09-14 03:20:29.356+0000: 29132: debug : virFileClose:72 : Closed fd 23 > 2012-09-14 03:20:29.356+0000: 29132: debug : virCgroupSetValueStr:335 : > Failed to write value '1': Device or resource busy > 2012-09-14 03:20:29.356+0000: 29132: error : > qemuDomainSetNumaParameters:7549 : unable to set numa tunable: Unknown error > 18446744073709551615 Looks like we should the useful error "Device or resource busy". commit a71ec98841c825dfb7e9b648dd85f5140f29d090
Author: Osier Yang <jyang>
Date: Mon Apr 22 19:40:19 2013 +0800
qemu: Fix the wrong expression
Wrong use of the parentheses causes "rc" always having a boolean value,
either "1" or "0", and thus we can't get the detailed error message
when it fails:
Before (I only have 1 node):
% virsh numatune f18 --nodeset 12
error: Unable to change numa parameters
error: unable to set numa tunable: Unknown error -1
After:
virsh numatune f18 --nodeset 12
error: Unable to change numa parameters
error: unable to set numa tunable: Invalid argument
And see https://bugzilla.redhat.com/show_bug.cgi?id=920406#c7
We decided not to rebase libvirt in RHEL 6.5 to avoid stability issues we faced in 6.4. This bug has already been trivially fixed upstream but it is not considered important enough to be backported to RHEL 6.5. Thus I'm pushing this bug to RHEL 6.6 (and setting Upstream keyword to indicate we have patches upstream) for now. If you don't agree with this resolution, please, give us reasons which you think are strong enough for us to reevaluate the decision not to backport patches for this bug. The commit in comment #9 only fixed the error message, the functionality has been fixed upstream by: commit 81fae6b95cfe72d0f5a987b6b5cd4bf86e32798c Author: Shivaprasad G Bhat <shivaprasadbhat> AuthorDate: 2013-12-03 06:30:26 -0500 Commit: Martin Kletzander <mkletzan> CommitDate: 2013-12-04 13:32:07 +0100 qemu: fix live pinning to memory node on NUMA system Ever since the subcpusets(vcpu,emulator) were introduced, the parent cpuset cannot be modified to remove the nodes that are in use by the subcpusets. The fix is to break the memory node modification into three steps: 1. assign new nodes into the parent, 2. change the nodes in the child nodes, 3. remove the old nodes on the parent node. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1009880 Signed-off-by: Shivaprasad G Bhat <sbhat.ibm.com> Signed-off-by: Martin Kletzander <mkletzan> git describe: v1.2.0-43-g81fae6b contains: v1.2.1-rc1~208 I can reproduce it on libvirt-0.10.2-29.el6.x86_64, but can not reproduce it on below version:
Version:
libvirt-0.10.2-32.el6.x86_64
qemu-kvm-0.12.1.2-2.415.el6.x86_64
Steps:
[root@hp-dl585g7-01 ~]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 4
NUMA node(s): 8
Vendor ID: AuthenticAMD
CPU family: 21
Model: 1
Stepping: 2
CPU MHz: 2593.567
BogoMIPS: 5186.46
Virtualization: AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache: 2048K
L3 cache: 6144K
NUMA node0 CPU(s): 0,4,8,12,16,20,24,28
NUMA node1 CPU(s): 32,36,40,44,48,52,56,60
NUMA node2 CPU(s): 1,5,9,13,17,21,25,29
NUMA node3 CPU(s): 33,37,41,45,49,53,57,61
NUMA node4 CPU(s): 2,6,10,14,18,22,26,30
NUMA node5 CPU(s): 34,38,42,46,50,54,58,62
NUMA node6 CPU(s): 35,39,43,47,51,55,59,63
NUMA node7 CPU(s): 3,7,11,15,19,23,27,31
[root@hp-dl585g7-01 ~]# virsh list --all
Id Name State
----------------------------------------------------
2 r6 running
[root@hp-dl585g7-01 ~]# virsh numatune r6
numa_mode : strict
numa_nodeset : 0
[root@hp-dl585g7-01 ~]# virsh numatune r6 --nodeset 0-3
[root@hp-dl585g7-01 ~]# virsh numatune r6
numa_mode : strict
numa_nodeset : 0-3
[root@hp-dl585g7-01 ~]# virsh numatune r6 --nodeset 8
error: Unable to change numa parameters
error: Failed to set cpuset.mems to 8: Invalid argument
[root@hp-dl585g7-01 ~]# virsh numatune r6 --nodeset 7
[root@hp-dl585g7-01 ~]# virsh numatune r6
numa_mode : strict
numa_nodeset : 7
[root@hp-dl585g7-01 ~]# virsh numatune r6 strict --nodeset 2 --current
[root@hp-dl585g7-01 ~]# virsh numatune r6
numa_mode : strict
numa_nodeset : 2
[root@hp-dl585g7-01 ~]# virsh numatune r6 strict --nodeset 1 --live
[root@hp-dl585g7-01 ~]# virsh numatune r6
numa_mode : strict
numa_nodeset : 1
[root@hp-dl585g7-01 ~]# virsh numatune r6 strict --nodeset 3 --config
[root@hp-dl585g7-01 ~]# virsh destroy r6
Domain r6 destroyed
[root@hp-dl585g7-01 ~]# virsh dumpxml r6 | grep numatune -A5
<numatune>
<memory mode='strict' nodeset='3'/>
</numatune>
<os>
<type arch='x86_64' machine='rhel6.5.0'>hvm</type>
<boot dev='network'/>
</os>
<features>
[root@hp-dl585g7-01 ~]# virsh numatune r6
numa_mode : strict
numa_nodeset : 3
[root@hp-dl585g7-01 ~]# virsh start r6
Domain r6 started
[root@hp-dl585g7-01 ~]# virsh numatune r6
numa_mode : strict
numa_nodeset : 3
On common PC(just one numa cell),version:
libvirt-0.10.2-32.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.423.el6.x86_64
[root@intel-5205-32-2 src]# virsh nodeinfo
CPU model: x86_64
CPU(s): 4
CPU frequency: 1861 MHz
CPU socket(s): 2
Core(s) per socket: 2
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 32875776 KiB
[root@intel-5205-32-2 src]# virsh list --all
Id Name State
----------------------------------------------------
15 r6 running
[root@intel-5205-32-2 src]# virsh numatune r6
numa_mode : strict
numa_nodeset : 0
[root@intel-5205-32-2 src]# virsh numatune r6 --nodeset 1
error: Unable to change numa parameters
error: Failed to set cpuset.mems to 1: Invalid argument
[root@intel-5205-32-2 src]# virsh numatune r6 --nodeset 0
[root@intel-5205-32-2 src]# virsh numatune r6
numa_mode : strict
numa_nodeset : 0
We can get expected results, changed to verified.
The patch from comment 22 is only related to the error message, so this reproduce step is focus on the error message: # rpm -q libvirt libvirt-0.10.2-39.el6.x86_64 # virsh start r6 # virsh numatune r6 --live numa_mode : strict numa_nodeset : 0 # umount /cgroup/cpuset/ # virsh numatune r6 --nodeset 0 --live error: Unable to change numa parameters error: out of memory Note, the error message is "out of memory", that is wrong message. In latest libvirt-0.10.2-40.el6, after umount cpuset, the error is: # virsh numatune r6 --nodeset 0 --live error: Unable to change numa parameters error: unable to get numa nodeset: No such file or directory This is right message, and I will change the status to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1374.html |