Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
+++ This bug was initially created as a clone of Bug #1173507 +++
Description of the problem
--------------------------
This occurred when booting a guest with a Nova flavor defined with huge pages size set to 'any' -- this is work in progress upstream[*] -- results in:
libvirtError: internal error: Unable to find any usable hugetlbfs mount for 4 KiB
From Nova Conductor logs:
. . .
2014-12-11 13:06:34.738 ERROR nova.scheduler.utils [req-7812c740-ec60-461e-a6b7-66b4bd4359ee admin admin] [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c
9fb] Error from last host: fedvm1 (node fedvm1): [u'Traceback (most recent call last):\n', u' File "/home/kashyapc/src/cloud/nova/nova/compute/manage
r.py", line 2060, in _do_build_and_run_instance\n filter_properties)\n', u' File "/home/kashyapc/src/cloud/nova/nova/compute/manager.py", line 220
0, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance c8e1093b-81d6-4
bc8-a319-7a8ea384c9fb was re-scheduled: internal error: Unable to find any usable hugetlbfs mount for 4 KiB\n']
. . .
[*] http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/virt-driver-large-pages.html#proposed-change
Version
-------
Apply the virt-driver-large-pages patch series to Nova git, and test via
DevStack:
https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/virt-driver-large-pages,n,z
$ git log | grep "commit\ " | head -8
commit c0c5d6a497c0e275e6f2037c1f7d45983a077cbc
commit 9d1d59bd82a7f2747487884d5880270bfdc9734a
commit eda126cce41fd5061b630a1beafbf5c37292946e
commit 6980502683bdcf514b386038ca0e0ef8226c27ca
commit b1ddc34efdba271f406a6db39c8deeeeadcb8cc9
This commit also add a new exceptions MemoryPageSizeInvalid and
commit 2fcfc675aa04ef2760f0e763697c73b6d90a4fca
commit 567987035bc3ef685ea09ac2b82be55aa5e23ca5
$ git describe
2014.2-1358-gc0c5d6a
libvirt version: libvirt-1.2.11 (built from libvirt git)
$ git log | head -1
commit a2a35d0164f4244b9c6f143f54e9bb9f3c9af7d3a
$ git describe
CVE-2014-7823-247-ga2a35d0
Steps to Reproduce
------------------
Test environment: I was testing Nova huge pages in a DevStack VM with KVM
nested virtualization, i.e. the Nova instances will be the nested guests.
Check if the 'hugetlbfs' is present in /proc filesystem:
$ cat /proc/filesystems | grep hugetlbfs
nodev hugetlbfs
Get the number of total huge pages:
$ grep HugePages_Total /proc/meminfo
HugePages_Total: 512
Get the number of free huge pages:
$ grep HugePages_Free /proc/meminfo
HugePages_Free: 512
Create flavor:
nova flavor-create m1.hugepages 999 2048 1 4
Set extra_spec values for NUMA and Huge pages, with value as 'any':
nova flavor-key m1.hugepages set hw:numa_nodes=1
nova flavor-key m1.hugepages set hw:mem_page_size=any
Enumerate the newly created flavor properties:
$ nova flavor-show m1.hugepages
+----------------------------+-----------------------------------------------------+
| Property | Value |
+----------------------------+-----------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 1 |
| extra_specs | {"hw:mem_page_size": "any", "hw:numa_nodes": "1"} |
| id | 999 |
| name | m1.hugepages |
| os-flavor-access:is_public | True |
| ram | 2048 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 4 |
+----------------------------+-----------------------------------------------------+
Boot a guest with the above falvor:
Actual results
--------------
(1) Contextual error messages from Nova Compute log (screen-n-cpu.log):
. . .
2014-12-11 13:06:34.141 ERROR nova.compute.manager [-] [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] Instance failed to spawn
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] Traceback (most recent call last):
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2282, in _build_resources
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] yield resources
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2152, in _build_and_run_instance
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] flavor=flavor)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/home/kashyapc/src/cloud/nova/nova/virt/li
bvirt/driver.py", line 2384, in spawn
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] block_device_info=block_device_info)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4278, in _create_domain_and_network
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] power_on=power_on)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4211, in _create_domain
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] LOG.error(err)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/usr/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] six.reraise(self.type_, self.value, self.tb)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4201, in _create_domain
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] domain.createWithFlags(launch_flags)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] rv = execute(f, *args, **kwargs)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] six.reraise(c, e, tb)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] rv = meth(*args, **kwargs)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1033, in createWithFlags
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-12-11 13:06:34.141 TRACE nova.compute.manager [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c9fb] libvirtError: internal error: Unable to find any usable hugetlbfs mount for 4 KiB
. . .
Expected results
----------------
As specified in the SPEC, Compute driver
Additional info
---------------
(2) Contextual error messages from Nova Conductor log (screen-n-cond.log):
----------------------------------------
. . .
2014-12-11 13:06:34.738 ERROR nova.scheduler.utils [req-7812c740-ec60-461e-a6b7-66b4bd4359ee admin admin] [instance: c8e1093b-81d6-4bc8-a319-7a8ea384c
9fb] Error from last host: fedvm1 (node fedvm1): [u'Traceback (most recent call last):\n', u' File "/home/kashyapc/src/cloud/nova/nova/compute/manage
r.py", line 2060, in _do_build_and_run_instance\n filter_properties)\n', u' File "/home/kashyapc/src/cloud/nova/nova/compute/manager.py", line 220
0, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance c8e1093b-81d6-4
bc8-a319-7a8ea384c9fb was re-scheduled: internal error: Unable to find any usable hugetlbfs mount for 4 KiB\n']
. . .
----------------------------------------
(3) This error comes from libvirt, added in this commit:
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=281f70013e
--- Additional comment from Kashyap Chamarthy on 2014-12-12 05:30:41 EST ---
(In reply to Kashyap Chamarthy from comment #0)
[. . .]
>
> Expected results
> ----------------
>
> As specified in the SPEC, Compute driver
Broken sentence ^^ . I was referring to the OpenStack Nova spec[1].
In this case, expected results: If huge pages are present, libvirt driver should find them, if not fallback to small pages"
[1] http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/virt-driver-large-pages.html#proposed-change
Additional info
---------------
Posting an email comment from Michal Privoznik:
This is a libvirt bug. What my patches allowed is to specify the huge
pages size to be used either per whole domain or per domain's NUMA
node. When writing the patch I agreed with Dan that it's gonna be much
more easier for OpenStack if libvirt would treat regular system pages
and huge pages the same. Obviously, it isn't doing that. Although one
may argue, that <hugepages/> element should accept huge pages only, it
will make life easier for you guys if it would accept system pages
too. Patch on its way. @Kayshap, Can you please open a bug
--- Additional comment from Kashyap Chamarthy on 2014-12-12 05:47:39 EST ---
--- Additional comment from Kashyap Chamarthy on 2014-12-12 05:51:17 EST ---
--- Additional comment from Kashyap Chamarthy on 2014-12-13 04:55:33 EST ---
Upstream patch by Michal Privoznik
http://www.redhat.com/archives/libvir-list/2014-December/msg00669.html
--- Additional comment from Michal Privoznik on 2014-12-15 10:47:23 EST ---
I've pushed patch upstream:
commit 311b4a677f60cc1a3a29c525a703b31ec47d95b5
Author: Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 12 10:37:35 2014 +0100
Commit: Michal Privoznik <mprivozn>
CommitDate: Mon Dec 15 13:36:47 2014 +0100
qemu: Allow system pages to <memoryBacking/>
https://bugzilla.redhat.com/show_bug.cgi?id=1173507
It occurred to me that OpenStack uses the following XML when not using
regular huge pages:
<memoryBacking>
<hugepages>
<page size='4' unit='KiB'/>
</hugepages>
</memoryBacking>
However, since we are expecting to see huge pages only, we fail to
startup the domain with following error:
libvirtError: internal error: Unable to find any usable hugetlbfs
mount for 4 KiB
While regular system pages are not huge pages technically, our code is
prepared for that and if it helps OpenStack (or other management
applications) we should cope with that.
Signed-off-by: Michal Privoznik <mprivozn>
v1.2.11-12-g311b4a6
Btw, looking at the current version (47) of openstack patches at https://review.openstack.org/#/c/128703/47/nova/virt/libvirt/driver.py I don't see any need to backport the libvirt commit anywhere. In the current version openstack will not try to use normal pages in /domain/memoryBacking/hugepages and they need to keep that code there until the minimal required version of libvirt is 1.2.12. In other words, I suggest closing this as WONTFIX.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2015-2202.html