Description of problem:
Libvirt fails to pin a guest to a specifig memory node at runtime or at guest creation.

Version-Release number of selected component (if applicable):
Libvirt 1.1.1
Linux 3.2.51
Linux 3.11.1

How reproducible:
Always

Steps to Reproduce:
1. Start a guest on a NUMA system without any CPU or memory pinning
2. Pin it to the second node: # virsh numatune test --nodeset 1

Actual results:
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine/test.libvirt-
qemu/cpuset.mems': Device or resource busy

Expected results:
Succesfull pinning to the second node

Additional info:
I have a theory of what could be the problem:
Lets assume we have a 2 node dualcore system and start a guest named 'test' 
without cpu or memory pinning.

* libvirt creates a controller under cpuset/machine/test.libvirt-qemu:
cpuset/machine/test.libvirt-qemu/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/cpuset.mems -> 0-1
* libvirt creates a controller for every vcpu:
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.mems -> 0-1
* libvirt creates a controller for qemu:
cpuset/machine/test.libvirt-qemu/emulator/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/emulator/cpuset.mems -> 0-1

Now we want to pin the guest to the second node
virsh # numatune test --nodeset 1 
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine/test.libvirt-
qemu/cpuset.mems': Device or resource busy

What happens is that Libvirt tries to set cpuset/machine/test.libvirt-
qemu/cpuset.mems to 1 but this is not possible because 
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.mems and 
cpuset/machine/test.libvirt-qemu/emulator/cpuset.mems still contain 0-1.
Libvirt has to change these values before!

If I do this manually, numatune runs fine:
# echo 1 > vcpu0/cpuset.mems 
# echo 1 > emulator/cpuset.mems
# virsh numatune test --nodeset 1

I have tested this with kernel 3.2.51 and 3.11.1 on a Gentoo system.


Cheers,
David