Bug 515285
| Summary: | RFE: support configuration of hugepages for guests | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Daniel Berrangé <berrange> | ||||
| Component: | libvirt | Assignee: | Daniel Veillard <veillard> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 6.0 | CC: | ajia, berrange, crobinso, hbrock, jialiu, jjarvis, kmonroe, nelhawar, syeghiay, troels, veillard, virt-maint, xen-maint | ||||
| Target Milestone: | beta | Keywords: | FutureFeature | ||||
| Target Release: | 6.0 | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-0.7.1-2.el6.1 | Doc Type: | Enhancement | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | 465532 | Environment: | |||||
| Last Closed: | 2010-07-02 19:24:49 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 537389 | ||||||
| Bug Blocks: | 356741, 554559 | ||||||
| Attachments: |
|
||||||
|
Description
Daniel Berrangé
2009-08-03 15:34:33 UTC
cf initial patches from John Cooper http://www.redhat.com/archives/libvir-list/2009-July/msg00753.html This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux major release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Major release. This request is not yet committed for inclusion. IBM is also interested in seeing the feature in RHEL 6 ------- Comment From ratliff.com 2009-08-18 14:58 EDT------- 1. Feature Overview: Feature Id: [202016] a. Name of Feature: libvirt control hugepages b. Feature Description Have libvirt provide controls over the use of hugepages to back VMs 2. Feature Details: Sponsor: LTC Architectures: x86 x86_64 Arch Specificity: Both Delivery Mechanism: Direct from community Category: Xen Request Type: Package - Feature from Upstream d. Upstream Acceptance: In Progress Sponsor Priority 2 f. Severity: Medium IBM Confidential: no Code Contribution: 3rd party code g. Component Version Target: latest version of libvirt 3. Business Case On systems with hardware paging support (EPT/NPT) using largepages through hugetlbfs to back the VM can provide a significant perforamce improvement for some workloads. Currently this isn't exposed through libvirt as a guest configurable option. Add support for this will allow managemnt software greater flexibility to control quest config in performance sensitive areas. 4. Primary contact at Red Hat: John Jarvis jjarvis 5. Primary contacts at Partner: Project Management Contact: Stephanie Glass, sglass.com, 512-838-9284 Technical contact(s): Ryan Harper, raharper.com IBM Manager: Warren Grunbok II, grunbok.com An updated patch has been submitted upstream with greater capabilities http://www.redhat.com/archives/libvir-list/2009-August/msg00480.html This was included in the 0.7.1 release of libvirt Now built in RHEL-6 version libvirt-0.7.1-2.el6.1 Fixed in 'libvirt-0.7.1-2.el6.1'. 'libvirt-0.7.1-2.el6.2' included in compose 'RHEL6.0-20091029.0'. Moving to ON_QA. Created attachment 368341 [details]
Domain config xml file
Test this defect with libvirt-0.7.1-2.el6.2 on x86_64, but failed.
[root@dhcp-66-70-119 ~]# uname -a
Linux dhcp-66-70-119.nay.redhat.com 2.6.31-33.el6.x86_64 #1 SMP Thu Oct 29 12:00
:41 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@dhcp-66-70-119 ~]# more /etc/redhat-release
Red Hat Enterprise Linux release 6.0 Alpha (Santiago)
[root@dhcp-66-70-119 ~]# lsmod |grep kvm
kvm_amd 45056 0
kvm 192512 1 kvm_amd
[root@dhcp-66-70-119 ~]# rpm -qa|grep libvirt
libvirt-0.7.1-2.el6.2.x86_64
libvirt-client-0.7.1-2.el6.2.x86_64
libvirt-python-0.7.1-2.el6.2.x86_64
libvirt-devel-0.7.1-2.el6.2.x86_64
libvirt-cim-0.5.6-2.el6.x86_64
=============Configure guest with hugepage support via libvirt============
domain config xml file:
...
...
...
<memoryBacking>
<hugepages/>
</memoryBacking>
...
...
...
Note: Domain config xml file(rhel5.4_i386.xml) has been attached.
[root@dhcp-66-70-119 ~]# mkdir /dev/hugepages
[root@dhcp-66-70-119 ~]# mount -t hugetlbfs hugtlbfs /dev/hugepages
[root@dhcp-66-70-119 ~]# sysctl vm.nr_hugepages=516
vm.nr_hugepages = 516
[root@dhcp-66-70-119 ~]# cat /proc/mounts |grep hugtlb
hugtlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
[root@dhcp-66-70-119 ~]# service libvirtd restart
Stopping libvirtd daemon: [ OK ]
Starting libvirtd daemon: [ OK ]
[root@dhcp-66-70-119 ~]# virsh start rhel5.4_i386
Domain rhel5.4_i386 started
[root@dhcp-66-70-119 ~]# ps -ef|grep qemu
qemu 2771 1 43 02:39 ? 00:00:45 /usr/bin/qemu-kvm -S -M pc-0.11
-cpu qemu32 -m 0 -mem-path /dev/hugepages/libvirt/qemu -smp 1 -name rhel5.4_i386
-uuid 1cd779e0-5bd9-e8bb-0b9b-4d4bedd28a8b -monitor unix:/var/lib/libvirt/qemu/
rhel5.4_i386.monitor,server,nowait -boot c -drive file=/var/lib/libvirt/images/r
hel5.4_i386.img,if=ide,index=0,boot=on,format=raw -drive file=,if=ide,media=cdro
m,index=2 -net nic,macaddr=52:54:00:26:dd:a4,vlan=0,name=nic.0 -net tap,fd=16,vl
an=0,name=tap.0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga c
irrus
NOTE: Seem hugepage support has been passed using "-mem-path" by libvirt.
[root@dhcp-66-70-119 ~]# more /proc/meminfo |grep Hug
HugePages_Total: 516
HugePages_Free: 516
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
NOTE: Looks like the hugepage is never used, so test failed.
========start kvm guest with hugepage support directly using qemu-kvm=========
[root@dhcp-66-70-119 ~]# qemu-kvm -hda /var/lib/libvirt/images/rhel5.4_i386.img
-name aaa -m 100 -mem-path /dev/hugepages/libvirt/qemu
[root@dhcp-66-70-119 ~]# more /proc/meminfo |grep Hug
HugePages_Total: 516
HugePages_Free: 458
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
NOTE: If start kvm guest directly using qemu-kvm command with "-mem-path" parameter, hugepage support works fine.
I know why I still reproduced the defect, because in my steps, there is a typo in the command that I typed. "mount -t hugetlbfs hugtlbfs /dev/hugepages" should be correct to "mount -t hugetlbfs hugetlbfs /dev/hugepages" After I correct the typo, hugepage support for kvm guest works fine. Sorry for the inconvenience. I checked the defect again. For libvirt-0.7.1-2.el6.2 package, if selinux is turn on, the test failed as comment #12; if selinux is turn off, the test will passed. So the root cause of my test fail is selinux, so please ignore the comment #13. Confirmed that this is SELinux policy denying use of hugepages, and opened this bug to track resolution of that problem https://bugzilla.redhat.com/show_bug.cgi?id=537389 ------- Comment From agl.com 2010-06-07 16:55 EDT------- The SELinux policy has been corrected in Snap6. Marking as Verified. Verified this defect with libvirt-0.8.1-10.el6.x86_64 on nightly build (RHEL6.0-20100622.n.0), and PASSED.
Steps:
1. #mkdir /dev/hugepages
2. #mount -t hugetlbfs hugetlbfs /dev/hugepages
3. #sysctl vm.nr_hugepages=100
4. #service libvirtd restart
5. cat meminfo before guest start
# more /proc/meminfo |grep Huge
AnonHugePages: 16384 kB
HugePages_Total: 100
HugePages_Free: 100
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
6. Run a guest with the following xml added to domain xml.
<memoryBacking><hugepages/></memoryBacking>
7. Start the domain.
8. Check HugePages_Free in meminfo again, confirm the guest is using hugepages.
# more /proc/meminfo |grep Huge
AnonHugePages: 86016 kB
HugePages_Total: 100
HugePages_Free: 99
HugePages_Rsvd: 7
HugePages_Surp: 0
Hugepagesize: 2048 kB
Red Hat Enterprise Linux Beta 2 is now available and should resolve the problem described in this bug report. This report is therefore being closed with a resolution of CURRENTRELEASE. You may reopen this bug report if the solution does not work for you. |