Bug 515285 - RFE: support configuration of hugepages for guests
Summary: RFE: support configuration of hugepages for guests
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.0
Hardware: All
OS: Linux
Target Milestone: beta
: 6.0
Assignee: Daniel Veillard
QA Contact: Virtualization Bugs
Depends On: 537389
Blocks: 356741 554559
TreeView+ depends on / blocked
Reported: 2009-08-03 15:34 UTC by Daniel Berrangé
Modified: 2011-04-12 16:06 UTC (History)
13 users (show)

Fixed In Version: libvirt-0.7.1-2.el6.1
Doc Type: Enhancement
Doc Text:
Clone Of: 465532
Last Closed: 2010-07-02 19:24:49 UTC

Attachments (Terms of Use)
Domain config xml file (1.42 KB, text/plain)
2009-11-10 08:19 UTC, Johnny Liu
no flags Details

Description Daniel Berrangé 2009-08-03 15:34:33 UTC
+++ This bug was initially created as a clone of Bug #465532 +++

According to reports from KVM developers, backing a KVM virtual guest with hugetlbfs can improve the performance of guests up to 10%:


The steps described in the above link aren't too difficult, but at the minimum, virt-manager should support passing –mem-path to qemu-kvm (e.g., allowing you to specify the path to the hugetlbfs to use).

(It would be super cool and spiffy if virt-manager were to allow the user to setup the hugetlbfs automagically, but that's a lot more complex, and would probably require a reboot. Being able to support using an existing hugetlbfs by specifying its path would be a reasonable compromise and should be easy to implement.)

--- Additional comment from berrange@redhat.com on 2008-10-03 15:31:52 EDT ---

I definitely do not want to expose the -mem-path implementation detail to users or virt-manager. They simply do not have the knowledge neccessary to understand what this means, nor how to leverage it. If we're to make use of this capability we need to figure out a better way to handle it in libvirt and/or kvm, preferably such that it 'just works' as it does in Xen world without any admin config.

Changing to libvirt, because this definitely isn't the job of virt-manager

--- Additional comment from ralston@pobox.com on 2008-10-06 21:55:38 EDT ---

Well, the thing about hugepages is, you pretty much have to allocate them at system startup.  But memory reserved for hugepages can't be used normally by regular processes.

Also, it's not clear that virtualization could be the exclusive user of hugepages; there might be other things that could want to use hugepages.

Honestly, I think to *really* do this right, there needs to be some sort of system startup hook to allocate hugepages and mount hugetlsfs in a standard location (e.g., /hugepages).  There also needs to be some sort of configuration program (e.g., system-config-memory) to set the amount of memory allocated as hugepages.  (Maybe that could be folder into an existing utility.)

Then, when starting a virtual guest, libvirt could inspect /proc/meminfo to see if enough non-reserved hugepages are available to satisfy the virtual guest's memory size.  If yes, then libvirt automatically includes –mem-path on the command line; if not, then it doesn't.

Optionally, libvirt could expose tunables to virt-manager, so that users could, say, fine-tune which virtual guests grab hugepages, limit the number of hugepages that libvirt will attempt to grab (e.g., if other processes in the system were also wanting to use hugepages), et. al.


--- Additional comment from nelhawar@redhat.com on 2008-11-26 01:33:17 EDT ---

This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:

Comment 1 Daniel Berrangé 2009-08-03 15:35:11 UTC
cf initial patches from John Cooper


Comment 3 RHEL Product and Program Management 2009-08-03 15:59:35 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for

Comment 4 IBM Bug Proxy 2009-08-19 17:42:50 UTC
 IBM is also interested in seeing the feature in RHEL 6

Comment 5 IBM Bug Proxy 2009-08-19 17:54:27 UTC
------- Comment From ratliff@austin.ibm.com 2009-08-18 14:58 EDT-------
1. Feature Overview:
Feature Id:	[202016]
a. Name of Feature:	libvirt control hugepages
b. Feature Description
Have libvirt provide controls over the use of hugepages to back VMs

2. Feature Details:
Sponsor:	LTC

Arch Specificity: Both
Delivery Mechanism: Direct from community
Category:	Xen
Request Type:	Package - Feature from Upstream
d. Upstream Acceptance:	In Progress
Sponsor Priority	2
f. Severity: Medium
IBM Confidential:	no
Code Contribution:	3rd party code
g. Component Version Target:	latest version of libvirt

3. Business Case
On systems with hardware paging support (EPT/NPT) using largepages through hugetlbfs to back the VM can provide a significant perforamce improvement for some workloads.  Currently this isn't exposed through libvirt as a guest configurable option.  Add support for this will allow managemnt software greater flexibility to control quest config in performance sensitive areas.

4. Primary contact at Red Hat:
John Jarvis

5. Primary contacts at Partner:
Project Management Contact:
Stephanie Glass, sglass@us.ibm.com, 512-838-9284

Technical contact(s):
Ryan Harper, raharper@us.ibm.com

IBM Manager:
Warren Grunbok II, grunbok@us.ibm.com

Comment 6 Daniel Berrangé 2009-08-26 13:06:53 UTC
An updated patch has been submitted upstream with greater capabilities


Comment 8 Daniel Berrangé 2009-09-17 12:06:17 UTC
This was included in the 0.7.1 release of libvirt

Comment 9 Daniel Berrangé 2009-09-17 17:40:26 UTC
Now built in RHEL-6 version libvirt-0.7.1-2.el6.1

Comment 10 releng-rhel@redhat.com 2009-10-30 22:12:23 UTC
Fixed in 'libvirt-0.7.1-2.el6.1'. 'libvirt-0.7.1-2.el6.2' included in compose 'RHEL6.0-20091029.0'.
Moving to ON_QA.

Comment 11 Johnny Liu 2009-11-10 08:19:09 UTC
Created attachment 368341 [details]
Domain config xml file

Comment 12 Johnny Liu 2009-11-10 08:46:58 UTC
Test this defect with libvirt-0.7.1-2.el6.2 on x86_64, but failed.

[root@dhcp-66-70-119 ~]# uname -a
Linux dhcp-66-70-119.nay.redhat.com 2.6.31-33.el6.x86_64 #1 SMP Thu Oct 29 12:00
:41 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@dhcp-66-70-119 ~]# more /etc/redhat-release 
Red Hat Enterprise Linux release 6.0 Alpha (Santiago)
[root@dhcp-66-70-119 ~]# lsmod |grep kvm
kvm_amd                45056  0 
kvm                   192512  1 kvm_amd
[root@dhcp-66-70-119 ~]# rpm -qa|grep libvirt

=============Configure guest with hugepage support via libvirt============
domain config xml file:

Note: Domain config xml file(rhel5.4_i386.xml) has been attached.

[root@dhcp-66-70-119 ~]# mkdir /dev/hugepages
[root@dhcp-66-70-119 ~]# mount -t hugetlbfs hugtlbfs /dev/hugepages
[root@dhcp-66-70-119 ~]# sysctl vm.nr_hugepages=516
vm.nr_hugepages = 516
[root@dhcp-66-70-119 ~]# cat /proc/mounts |grep hugtlb
hugtlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
[root@dhcp-66-70-119 ~]# service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]
[root@dhcp-66-70-119 ~]# virsh start rhel5.4_i386
Domain rhel5.4_i386 started
[root@dhcp-66-70-119 ~]# ps -ef|grep qemu
qemu      2771     1 43 02:39 ?        00:00:45 /usr/bin/qemu-kvm -S -M pc-0.11 
-cpu qemu32 -m 0 -mem-path /dev/hugepages/libvirt/qemu -smp 1 -name rhel5.4_i386
 -uuid 1cd779e0-5bd9-e8bb-0b9b-4d4bedd28a8b -monitor unix:/var/lib/libvirt/qemu/
rhel5.4_i386.monitor,server,nowait -boot c -drive file=/var/lib/libvirt/images/r
hel5.4_i386.img,if=ide,index=0,boot=on,format=raw -drive file=,if=ide,media=cdro
m,index=2 -net nic,macaddr=52:54:00:26:dd:a4,vlan=0,name=nic.0 -net tap,fd=16,vl
an=0,name=tap.0 -serial pty -parallel none -usb -vnc -k en-us -vga c

NOTE: Seem hugepage support has been passed using "-mem-path" by libvirt.

[root@dhcp-66-70-119 ~]# more /proc/meminfo |grep Hug
HugePages_Total:     516
HugePages_Free:      516
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

NOTE: Looks like the hugepage is never used, so test failed.

========start kvm guest with hugepage support directly using qemu-kvm=========
[root@dhcp-66-70-119 ~]# qemu-kvm -hda /var/lib/libvirt/images/rhel5.4_i386.img 
-name aaa -m 100 -mem-path /dev/hugepages/libvirt/qemu
[root@dhcp-66-70-119 ~]# more /proc/meminfo |grep Hug
HugePages_Total:     516
HugePages_Free:      458
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

NOTE: If start kvm guest directly using qemu-kvm command with "-mem-path" parameter, hugepage support works fine.

Comment 13 Johnny Liu 2009-11-12 08:10:49 UTC
I know why I still reproduced the defect, because in my steps, there is a typo in the command that I typed. 

"mount -t hugetlbfs hugtlbfs /dev/hugepages"  should be correct to "mount -t hugetlbfs hugetlbfs /dev/hugepages"

After I correct the typo, hugepage support for kvm guest works fine.

Sorry for the inconvenience.

Comment 14 Johnny Liu 2009-11-12 09:23:44 UTC
I checked the defect again. 
For libvirt-0.7.1-2.el6.2 package, if selinux is turn on, the test failed as comment #12; if selinux is turn off, the test will passed.

So the root cause of my test fail is selinux, so please ignore the comment #13.

Comment 15 Daniel Berrangé 2009-11-13 12:50:19 UTC
Confirmed that this is SELinux policy denying use of hugepages, and opened this bug to track resolution of that problem


Comment 16 IBM Bug Proxy 2010-06-07 21:00:56 UTC
------- Comment From agl@us.ibm.com 2010-06-07 16:55 EDT-------
The SELinux policy has  been corrected in Snap6.  Marking as Verified.

Comment 17 Johnny Liu 2010-06-24 13:31:54 UTC
Verified this defect with libvirt-0.8.1-10.el6.x86_64 on nightly build (RHEL6.0-20100622.n.0), and PASSED.

1. #mkdir /dev/hugepages
2. #mount -t hugetlbfs hugetlbfs /dev/hugepages
3. #sysctl vm.nr_hugepages=100
4. #service libvirtd restart
5. cat meminfo before guest start
# more /proc/meminfo |grep Huge
AnonHugePages:     16384 kB
HugePages_Total:     100
HugePages_Free:      100
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
6. Run a guest with the following xml added to domain xml.
7. Start the domain.
8. Check HugePages_Free in meminfo again, confirm the guest is using hugepages.
# more /proc/meminfo |grep Huge
AnonHugePages:     86016 kB
HugePages_Total:     100
HugePages_Free:       99
HugePages_Rsvd:        7
HugePages_Surp:        0
Hugepagesize:       2048 kB

Comment 18 releng-rhel@redhat.com 2010-07-02 19:24:49 UTC
Red Hat Enterprise Linux Beta 2 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.