Bug 515285 - RFE: support configuration of hugepages for guests
RFE: support configuration of hugepages for guests
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
All Linux
medium Severity medium
: beta
: 6.0
Assigned To: Daniel Veillard
Virtualization Bugs
: FutureFeature
Depends On: 537389
Blocks: 356741 554559
  Show dependency treegraph
Reported: 2009-08-03 11:34 EDT by Daniel Berrange
Modified: 2011-04-12 12:06 EDT (History)
13 users (show)

See Also:
Fixed In Version: libvirt-0.7.1-2.el6.1
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 465532
Last Closed: 2010-07-02 15:24:49 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Domain config xml file (1.42 KB, text/plain)
2009-11-10 03:19 EST, Johnny Liu
no flags Details

  None (edit)
Description Daniel Berrange 2009-08-03 11:34:33 EDT
+++ This bug was initially created as a clone of Bug #465532 +++

According to reports from KVM developers, backing a KVM virtual guest with hugetlbfs can improve the performance of guests up to 10%:


The steps described in the above link aren't too difficult, but at the minimum, virt-manager should support passing –mem-path to qemu-kvm (e.g., allowing you to specify the path to the hugetlbfs to use).

(It would be super cool and spiffy if virt-manager were to allow the user to setup the hugetlbfs automagically, but that's a lot more complex, and would probably require a reboot. Being able to support using an existing hugetlbfs by specifying its path would be a reasonable compromise and should be easy to implement.)

--- Additional comment from berrange@redhat.com on 2008-10-03 15:31:52 EDT ---

I definitely do not want to expose the -mem-path implementation detail to users or virt-manager. They simply do not have the knowledge neccessary to understand what this means, nor how to leverage it. If we're to make use of this capability we need to figure out a better way to handle it in libvirt and/or kvm, preferably such that it 'just works' as it does in Xen world without any admin config.

Changing to libvirt, because this definitely isn't the job of virt-manager

--- Additional comment from ralston@pobox.com on 2008-10-06 21:55:38 EDT ---

Well, the thing about hugepages is, you pretty much have to allocate them at system startup.  But memory reserved for hugepages can't be used normally by regular processes.

Also, it's not clear that virtualization could be the exclusive user of hugepages; there might be other things that could want to use hugepages.

Honestly, I think to *really* do this right, there needs to be some sort of system startup hook to allocate hugepages and mount hugetlsfs in a standard location (e.g., /hugepages).  There also needs to be some sort of configuration program (e.g., system-config-memory) to set the amount of memory allocated as hugepages.  (Maybe that could be folder into an existing utility.)

Then, when starting a virtual guest, libvirt could inspect /proc/meminfo to see if enough non-reserved hugepages are available to satisfy the virtual guest's memory size.  If yes, then libvirt automatically includes –mem-path on the command line; if not, then it doesn't.

Optionally, libvirt could expose tunables to virt-manager, so that users could, say, fine-tune which virtual guests grab hugepages, limit the number of hugepages that libvirt will attempt to grab (e.g., if other processes in the system were also wanting to use hugepages), et. al.


--- Additional comment from nelhawar@redhat.com on 2008-11-26 01:33:17 EDT ---

This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
Comment 1 Daniel Berrange 2009-08-03 11:35:11 EDT
cf initial patches from John Cooper

Comment 3 RHEL Product and Program Management 2009-08-03 11:59:35 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
Comment 4 IBM Bug Proxy 2009-08-19 13:42:50 EDT
 IBM is also interested in seeing the feature in RHEL 6
Comment 5 IBM Bug Proxy 2009-08-19 13:54:27 EDT
------- Comment From ratliff@austin.ibm.com 2009-08-18 14:58 EDT-------
1. Feature Overview:
Feature Id:	[202016]
a. Name of Feature:	libvirt control hugepages
b. Feature Description
Have libvirt provide controls over the use of hugepages to back VMs

2. Feature Details:
Sponsor:	LTC

Arch Specificity: Both
Delivery Mechanism: Direct from community
Category:	Xen
Request Type:	Package - Feature from Upstream
d. Upstream Acceptance:	In Progress
Sponsor Priority	2
f. Severity: Medium
IBM Confidential:	no
Code Contribution:	3rd party code
g. Component Version Target:	latest version of libvirt

3. Business Case
On systems with hardware paging support (EPT/NPT) using largepages through hugetlbfs to back the VM can provide a significant perforamce improvement for some workloads.  Currently this isn't exposed through libvirt as a guest configurable option.  Add support for this will allow managemnt software greater flexibility to control quest config in performance sensitive areas.

4. Primary contact at Red Hat:
John Jarvis

5. Primary contacts at Partner:
Project Management Contact:
Stephanie Glass, sglass@us.ibm.com, 512-838-9284

Technical contact(s):
Ryan Harper, raharper@us.ibm.com

IBM Manager:
Warren Grunbok II, grunbok@us.ibm.com
Comment 6 Daniel Berrange 2009-08-26 09:06:53 EDT
An updated patch has been submitted upstream with greater capabilities

Comment 8 Daniel Berrange 2009-09-17 08:06:17 EDT
This was included in the 0.7.1 release of libvirt
Comment 9 Daniel Berrange 2009-09-17 13:40:26 EDT
Now built in RHEL-6 version libvirt-0.7.1-2.el6.1
Comment 10 releng-rhel@redhat.com 2009-10-30 18:12:23 EDT
Fixed in 'libvirt-0.7.1-2.el6.1'. 'libvirt-0.7.1-2.el6.2' included in compose 'RHEL6.0-20091029.0'.
Moving to ON_QA.
Comment 11 Johnny Liu 2009-11-10 03:19:09 EST
Created attachment 368341 [details]
Domain config xml file
Comment 12 Johnny Liu 2009-11-10 03:46:58 EST
Test this defect with libvirt-0.7.1-2.el6.2 on x86_64, but failed.

[root@dhcp-66-70-119 ~]# uname -a
Linux dhcp-66-70-119.nay.redhat.com 2.6.31-33.el6.x86_64 #1 SMP Thu Oct 29 12:00
:41 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@dhcp-66-70-119 ~]# more /etc/redhat-release 
Red Hat Enterprise Linux release 6.0 Alpha (Santiago)
[root@dhcp-66-70-119 ~]# lsmod |grep kvm
kvm_amd                45056  0 
kvm                   192512  1 kvm_amd
[root@dhcp-66-70-119 ~]# rpm -qa|grep libvirt

=============Configure guest with hugepage support via libvirt============
domain config xml file:

Note: Domain config xml file(rhel5.4_i386.xml) has been attached.

[root@dhcp-66-70-119 ~]# mkdir /dev/hugepages
[root@dhcp-66-70-119 ~]# mount -t hugetlbfs hugtlbfs /dev/hugepages
[root@dhcp-66-70-119 ~]# sysctl vm.nr_hugepages=516
vm.nr_hugepages = 516
[root@dhcp-66-70-119 ~]# cat /proc/mounts |grep hugtlb
hugtlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
[root@dhcp-66-70-119 ~]# service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]
[root@dhcp-66-70-119 ~]# virsh start rhel5.4_i386
Domain rhel5.4_i386 started
[root@dhcp-66-70-119 ~]# ps -ef|grep qemu
qemu      2771     1 43 02:39 ?        00:00:45 /usr/bin/qemu-kvm -S -M pc-0.11 
-cpu qemu32 -m 0 -mem-path /dev/hugepages/libvirt/qemu -smp 1 -name rhel5.4_i386
 -uuid 1cd779e0-5bd9-e8bb-0b9b-4d4bedd28a8b -monitor unix:/var/lib/libvirt/qemu/
rhel5.4_i386.monitor,server,nowait -boot c -drive file=/var/lib/libvirt/images/r
hel5.4_i386.img,if=ide,index=0,boot=on,format=raw -drive file=,if=ide,media=cdro
m,index=2 -net nic,macaddr=52:54:00:26:dd:a4,vlan=0,name=nic.0 -net tap,fd=16,vl
an=0,name=tap.0 -serial pty -parallel none -usb -vnc -k en-us -vga c

NOTE: Seem hugepage support has been passed using "-mem-path" by libvirt.

[root@dhcp-66-70-119 ~]# more /proc/meminfo |grep Hug
HugePages_Total:     516
HugePages_Free:      516
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

NOTE: Looks like the hugepage is never used, so test failed.

========start kvm guest with hugepage support directly using qemu-kvm=========
[root@dhcp-66-70-119 ~]# qemu-kvm -hda /var/lib/libvirt/images/rhel5.4_i386.img 
-name aaa -m 100 -mem-path /dev/hugepages/libvirt/qemu
[root@dhcp-66-70-119 ~]# more /proc/meminfo |grep Hug
HugePages_Total:     516
HugePages_Free:      458
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

NOTE: If start kvm guest directly using qemu-kvm command with "-mem-path" parameter, hugepage support works fine.
Comment 13 Johnny Liu 2009-11-12 03:10:49 EST
I know why I still reproduced the defect, because in my steps, there is a typo in the command that I typed. 

"mount -t hugetlbfs hugtlbfs /dev/hugepages"  should be correct to "mount -t hugetlbfs hugetlbfs /dev/hugepages"

After I correct the typo, hugepage support for kvm guest works fine.

Sorry for the inconvenience.
Comment 14 Johnny Liu 2009-11-12 04:23:44 EST
I checked the defect again. 
For libvirt-0.7.1-2.el6.2 package, if selinux is turn on, the test failed as comment #12; if selinux is turn off, the test will passed.

So the root cause of my test fail is selinux, so please ignore the comment #13.
Comment 15 Daniel Berrange 2009-11-13 07:50:19 EST
Confirmed that this is SELinux policy denying use of hugepages, and opened this bug to track resolution of that problem

Comment 16 IBM Bug Proxy 2010-06-07 17:00:56 EDT
------- Comment From agl@us.ibm.com 2010-06-07 16:55 EDT-------
The SELinux policy has  been corrected in Snap6.  Marking as Verified.
Comment 17 Johnny Liu 2010-06-24 09:31:54 EDT
Verified this defect with libvirt-0.8.1-10.el6.x86_64 on nightly build (RHEL6.0-20100622.n.0), and PASSED.

1. #mkdir /dev/hugepages
2. #mount -t hugetlbfs hugetlbfs /dev/hugepages
3. #sysctl vm.nr_hugepages=100
4. #service libvirtd restart
5. cat meminfo before guest start
# more /proc/meminfo |grep Huge
AnonHugePages:     16384 kB
HugePages_Total:     100
HugePages_Free:      100
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
6. Run a guest with the following xml added to domain xml.
7. Start the domain.
8. Check HugePages_Free in meminfo again, confirm the guest is using hugepages.
# more /proc/meminfo |grep Huge
AnonHugePages:     86016 kB
HugePages_Total:     100
HugePages_Free:       99
HugePages_Rsvd:        7
HugePages_Surp:        0
Hugepagesize:       2048 kB
Comment 18 releng-rhel@redhat.com 2010-07-02 15:24:49 EDT
Red Hat Enterprise Linux Beta 2 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.