RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1431793 - KVM-RT guests won't start anymore
Summary: KVM-RT guests won't start anymore
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Andrea Bolognani
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1175463
TreeView+ depends on / blocked
 
Reported: 2017-03-13 18:45 UTC by Luiz Capitulino
Modified: 2017-08-02 00:03 UTC (History)
11 users (show)

Fixed In Version: libvirt-3.2.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 17:24:15 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Luiz Capitulino 2017-03-13 18:45:57 UTC 
Description of problem:

KVM-RT guests don't start with libvirt 3.1.0, whereas they would start nomrally with libivrt 3.0.0. The reason is this commit:

  commit c2e60ad0e5124482942164e5fec088157f5e716a
  Author: Andrea Bolognani <abologna>
  Date:   Mon Feb 6 17:54:49 2017 +0100

      qemu: Forbid <memoryBacking><locked> without <memtune><hard_limit>

This commit makes <memtune><hard_limit> required when using
<memoryBacking><locked>. However, not using <memtune><hard_limit> used to work because of the following combination of factors:

1. Guest is backed by hugeTLB pages (which are not accounted as mlocked pages)
2. libvirt does an automatic mlock increase of 1GB when seeing <memoryBacking><locked> in the XML
3. QEMU allocation doesn't go over 1GB

Discussion is going on upstream about the best way to solve this and improve <memoryBacking><locked> support in libvirt:

https://www.redhat.com/archives/libvir-list/2017-March/msg00534.html

Until a decision is made and patches are posted, I recommend reverting c2e60ad0e51.


Version-Release number of selected component (if applicable): libvirt-3.1.0-2.el7.x86_64


How reproducible:


Steps to Reproduce:
1. Rollback to libvirt 3.0
2. Configure a guest as explained above
3. Start and use it
4. Upgrade to libvirt 3.1
5. Guest won't start anymore
Comment 2 Pei Zhang 2017-03-14 02:23:48 UTC 
Hi Luiz,

QE also found this issue. And now we do our testing by starting KVM-RT VM with adding hard_limit like below. I noticed you also mentioned this in kvm-rt@ list. So is this solution a workaround? And with this bug fix, we will still use the original XML configuration? Thanks.

  <memtune>
    <hard_limit unit='G'>system_memory_in_gb</hard_limit>
  </memtune>



Best Regards,
Pei
Comment 4 Luiz Capitulino 2017-03-14 12:48:57 UTC 
Yes, using <hard_limit> is a workaround. After this is fixed, the original XML should work again.
Comment 6 Andrea Bolognani 2017-03-23 18:18:08 UTC 
Fix posted upstream.

  https://www.redhat.com/archives/libvir-list/2017-March/msg01089.html
Comment 7 Luiz Capitulino 2017-03-23 20:38:56 UTC 
Awesome, if you provide us a test package, we can test it against KVM-RT setup.
Comment 10 Andrea Bolognani 2017-03-28 16:28:01 UTC 
The fix has been committed upstream and will be included in the
upcoming 3.2.0 release.

commit 868d043a090776e645c87124bb0dc73105a7d397
Author: Andrea Bolognani <abologna>
Date:   Wed Mar 22 13:44:45 2017 +0100

    process: Translate "unlimited" correctly
    
    The value we use internally to represent the lack of a memory
    locking limit, VIR_DOMAIN_MEMORY_PARAM_UNLIMITED, doesn't
    match the value setrlimit() and prlimit() use for the same
    purpose, RLIM_INFINITY, so we have to handle the translation
    ourselves.
    
    Partially-resolves: https://bugzilla.redhat.com/1431793

commit 7e667664d28f90bf6916604a55ebad7e2d85305b
Author: Andrea Bolognani <abologna>
Date:   Wed Mar 22 13:44:13 2017 +0100

    qemu: Fix memory locking limit calculation
    
    For guests that use <memoryBacking><locked>, our only option
    is to remove the memory locking limit altogether.
    
    Partially-resolves: https://bugzilla.redhat.com/1431793

commit 4b67e7a3779a695ad4460874ff6848e086ba08f3
Author: Andrea Bolognani <abologna>
Date:   Thu Mar 16 18:23:56 2017 +0100

    Revert "qemu: Forbid <memoryBacking><locked> without <memtune><hard_limit>"
    
    This reverts commit c2e60ad0e5124482942164e5fec088157f5e716a.
    
    Turns out this check is excessively strict: there are ways
    other than <memtune><hard_limit> to raise the memory locking
    limit for QEMU processes, one prominent example being
    tweaking /etc/security/limits.conf.
    
    Partially-resolves: https://bugzilla.redhat.com/1431793
Comment 12 Luyao Huang 2017-06-02 02:51:52 UTC 
Verify this bug with libvirt-3.2.0-7.el7.x86_64:

1. enable locked in memory backend without set hard limit:

# virsh dumpxml r7
<domain type='kvm' id='2'>
  <name>r7</name>
  <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid>
  <maxMemory slots='16' unit='KiB'>15243264</maxMemory>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>1024000</currentMemory>
  <memoryBacking>
    <locked/>
  </memoryBacking>


2. check qemu process memlock:

# prlimit -p `pidof qemu-kvm`
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0         0 blocks
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited blocks
LOCKS      max number of file locks held      unlimited unlimited 
MEMLOCK    max locked-in-memory address space unlimited unlimited bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                8192      8192 
NPROC      max number of processes               127505    127505 
RSS        max resident set size              unlimited unlimited pages
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals         127505    127505 
STACK      max stack size                       8388608 unlimited bytes


3. restart a guest with hard limit set in xml:

# virsh dumpxml r7
<domain type='kvm' id='4'>
  <name>r7</name>
  <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid>
  <maxMemory slots='16' unit='KiB'>15243264</maxMemory>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>1024000</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>3024000</hard_limit>
  </memtune>
  <memoryBacking>
    <locked/>
  </memoryBacking>


# prlimit -p `pidof qemu-kvm`
RESOURCE   DESCRIPTION                              SOFT       HARD UNITS
AS         address space limit                 unlimited  unlimited bytes
CORE       max core file size                          0          0 blocks
CPU        CPU time                            unlimited  unlimited seconds
DATA       max data size                       unlimited  unlimited bytes
FSIZE      max file size                       unlimited  unlimited blocks
LOCKS      max number of file locks held       unlimited  unlimited 
MEMLOCK    max locked-in-memory address space 3096576000 3096576000 bytes
MSGQUEUE   max bytes in POSIX mqueues             819200     819200 bytes
NICE       max nice prio allowed to raise              0          0 
NOFILE     max number of open files                 8192       8192 
NPROC      max number of processes                127505     127505 
RSS        max resident set size               unlimited  unlimited pages
RTPRIO     max real-time priority                      0          0 
RTTIME     timeout for real-time tasks         unlimited  unlimited microsecs
SIGPENDING max number of pending signals          127505     127505 
STACK      max stack size                        8388608  unlimited bytes
Comment 13 errata-xmlrpc 2017-08-01 17:24:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 14 errata-xmlrpc 2017-08-02 00:03:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Note You need to log in before you can comment on or make changes to this bug.