Bug 1009880

Summary:	Cannot pin guest to memory node on NUMA system
Product:	[Community] Virtualization Tools	Reporter:	David Weber <wb>
Component:	libvirt	Assignee:	Libvirt Maintainers <libvirt-maint>
Status:	CLOSED UPSTREAM	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	acathrow, mkletzan, shivaprasadbhat, wb
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-12-04 12:34:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Weber 2013-09-19 12:12:31 UTC

Description of problem:
Libvirt fails to pin a guest to a specifig memory node at runtime or at guest creation.

Version-Release number of selected component (if applicable):
Libvirt 1.1.1
Linux 3.2.51
Linux 3.11.1

How reproducible:
Always

Steps to Reproduce:
1. Start a guest on a NUMA system without any CPU or memory pinning
2. Pin it to the second node: # virsh numatune test --nodeset 1

Actual results:
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine/test.libvirt-
qemu/cpuset.mems': Device or resource busy

Expected results:
Succesfull pinning to the second node

Additional info:
I have a theory of what could be the problem:
Lets assume we have a 2 node dualcore system and start a guest named 'test' 
without cpu or memory pinning.

* libvirt creates a controller under cpuset/machine/test.libvirt-qemu:
cpuset/machine/test.libvirt-qemu/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/cpuset.mems -> 0-1
* libvirt creates a controller for every vcpu:
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.mems -> 0-1
* libvirt creates a controller for qemu:
cpuset/machine/test.libvirt-qemu/emulator/cpuset.cpus -> 0-3
cpuset/machine/test.libvirt-qemu/emulator/cpuset.mems -> 0-1

Now we want to pin the guest to the second node
virsh # numatune test --nodeset 1 
error: Unable to change numa parameters
error: Unable to write to '/sys/fs/cgroup/cpuset/machine/test.libvirt-
qemu/cpuset.mems': Device or resource busy

What happens is that Libvirt tries to set cpuset/machine/test.libvirt-
qemu/cpuset.mems to 1 but this is not possible because 
cpuset/machine/test.libvirt-qemu/vcpu*/cpuset.mems and 
cpuset/machine/test.libvirt-qemu/emulator/cpuset.mems still contain 0-1.
Libvirt has to change these values before!

If I do this manually, numatune runs fine:
# echo 1 > vcpu0/cpuset.mems 
# echo 1 > emulator/cpuset.mems
# virsh numatune test --nodeset 1

I have tested this with kernel 3.2.51 and 3.11.1 on a Gentoo system.


Cheers,
David

Comment 1 Martin Kletzander 2013-12-04 12:34:10 UTC

Fixed upstream with v1.2.0-43-g81fae6b:

commit 81fae6b95cfe72d0f5a987b6b5cd4bf86e32798c
Author: Shivaprasad G Bhat <shivaprasadbhat>
Date:   Tue Dec 3 06:30:26 2013 -0500

    qemu: fix live pinning to memory node on NUMA system