Bug 1009880 - Cannot pin guest to memory node on NUMA system
Cannot pin guest to memory node on NUMA system
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Libvirt Maintainers
Depends On:
  Show dependency treegraph
Reported: 2013-09-19 08:12 EDT by David Weber
Modified: 2013-12-04 07:34 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-12-04 07:34:10 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description David Weber 2013-09-19 08:12:31 EDT
Description of problem:
Libvirt fails to pin a guest to a specifig memory node at runtime or at guest creation.

Version-Release number of selected component (if applicable):
Libvirt 1.1.1
Linux 3.2.51
Linux 3.11.1

How reproducible:

Steps to Reproduce:
1. Start a guest on a NUMA system without any CPU or memory pinning
2. Pin it to the second node: # virsh numatune test --nodeset 1

Actual results:
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine/test.libvirt-
qemu/cpuset.mems': Device or resource busy

Expected results:
Succesfull pinning to the second node

Additional info:
I have a theory of what could be the problem:
Lets assume we have a 2 node dualcore system and start a guest named 'test' 
without cpu or memory pinning.

* libvirt creates a controller under cpuset/machine/test.libvirt-qemu:
cpuset/machine/test.libvirt-qemu/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/cpuset.mems -> 0-1
* libvirt creates a controller for every vcpu:
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.mems -> 0-1
* libvirt creates a controller for qemu:
cpuset/machine/test.libvirt-qemu/emulator/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/emulator/cpuset.mems -> 0-1

Now we want to pin the guest to the second node
virsh # numatune test --nodeset 1 
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine/test.libvirt-
qemu/cpuset.mems': Device or resource busy

What happens is that Libvirt tries to set cpuset/machine/test.libvirt-
qemu/cpuset.mems to 1 but this is not possible because 
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.mems and 
cpuset/machine/test.libvirt-qemu/emulator/cpuset.mems still contain 0-1.
Libvirt has to change these values before!

If I do this manually, numatune runs fine:
# echo 1 > vcpu0/cpuset.mems 
# echo 1 > emulator/cpuset.mems
# virsh numatune test --nodeset 1

I have tested this with kernel 3.2.51 and 3.11.1 on a Gentoo system.

Comment 1 Martin Kletzander 2013-12-04 07:34:10 EST
Fixed upstream with v1.2.0-43-g81fae6b:

commit 81fae6b95cfe72d0f5a987b6b5cd4bf86e32798c
Author: Shivaprasad G Bhat <shivaprasadbhat@gmail.com>
Date:   Tue Dec 3 06:30:26 2013 -0500

    qemu: fix live pinning to memory node on NUMA system

Note You need to log in before you can comment on or make changes to this bug.