Bug 1431793

Summary:	KVM-RT guests won't start anymore
Product:	Red Hat Enterprise Linux 7	Reporter:	Luiz Capitulino <lcapitulino>
Component:	libvirt	Assignee:	Andrea Bolognani <abologna>
Status:	CLOSED ERRATA	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.4	CC:	abologna, berrange, bhu, daolivei, dyuan, juzhang, lcapitulino, lhuang, pezhang, rbalakri, xuzhang
Target Milestone:	rc	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-3.2.0-1.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-01 17:24:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1175463

Description Luiz Capitulino 2017-03-13 18:45:57 UTC

Description of problem:

KVM-RT guests don't start with libvirt 3.1.0, whereas they would start nomrally with libivrt 3.0.0. The reason is this commit:

  commit c2e60ad0e5124482942164e5fec088157f5e716a
  Author: Andrea Bolognani <abologna>
  Date:   Mon Feb 6 17:54:49 2017 +0100

      qemu: Forbid <memoryBacking><locked> without <memtune><hard_limit>

This commit makes <memtune><hard_limit> required when using
<memoryBacking><locked>. However, not using <memtune><hard_limit> used to work because of the following combination of factors:

1. Guest is backed by hugeTLB pages (which are not accounted as mlocked pages)
2. libvirt does an automatic mlock increase of 1GB when seeing <memoryBacking><locked> in the XML
3. QEMU allocation doesn't go over 1GB

Discussion is going on upstream about the best way to solve this and improve <memoryBacking><locked> support in libvirt:

https://www.redhat.com/archives/libvir-list/2017-March/msg00534.html

Until a decision is made and patches are posted, I recommend reverting c2e60ad0e51.


Version-Release number of selected component (if applicable): libvirt-3.1.0-2.el7.x86_64


How reproducible:


Steps to Reproduce:
1. Rollback to libvirt 3.0
2. Configure a guest as explained above
3. Start and use it
4. Upgrade to libvirt 3.1
5. Guest won't start anymore

Comment 2 Pei Zhang 2017-03-14 02:23:48 UTC

Hi Luiz,

QE also found this issue. And now we do our testing by starting KVM-RT VM with adding hard_limit like below. I noticed you also mentioned this in kvm-rt@ list. So is this solution a workaround? And with this bug fix, we will still use the original XML configuration? Thanks.

  <memtune>
    <hard_limit unit='G'>system_memory_in_gb</hard_limit>
  </memtune>



Best Regards,
Pei

Comment 4 Luiz Capitulino 2017-03-14 12:48:57 UTC

Yes, using <hard_limit> is a workaround. After this is fixed, the original XML should work again.

Comment 6 Andrea Bolognani 2017-03-23 18:18:08 UTC

Fix posted upstream.

  https://www.redhat.com/archives/libvir-list/2017-March/msg01089.html

Comment 7 Luiz Capitulino 2017-03-23 20:38:56 UTC

Awesome, if you provide us a test package, we can test it against KVM-RT setup.

Comment 10 Andrea Bolognani 2017-03-28 16:28:01 UTC

The fix has been committed upstream and will be included in the
upcoming 3.2.0 release.

commit 868d043a090776e645c87124bb0dc73105a7d397
Author: Andrea Bolognani <abologna>
Date:   Wed Mar 22 13:44:45 2017 +0100

    process: Translate "unlimited" correctly
    
    The value we use internally to represent the lack of a memory
    locking limit, VIR_DOMAIN_MEMORY_PARAM_UNLIMITED, doesn't
    match the value setrlimit() and prlimit() use for the same
    purpose, RLIM_INFINITY, so we have to handle the translation
    ourselves.
    
    Partially-resolves: https://bugzilla.redhat.com/1431793

commit 7e667664d28f90bf6916604a55ebad7e2d85305b
Author: Andrea Bolognani <abologna>
Date:   Wed Mar 22 13:44:13 2017 +0100

    qemu: Fix memory locking limit calculation
    
    For guests that use <memoryBacking><locked>, our only option
    is to remove the memory locking limit altogether.
    
    Partially-resolves: https://bugzilla.redhat.com/1431793

commit 4b67e7a3779a695ad4460874ff6848e086ba08f3
Author: Andrea Bolognani <abologna>
Date:   Thu Mar 16 18:23:56 2017 +0100

    Revert "qemu: Forbid <memoryBacking><locked> without <memtune><hard_limit>"
    
    This reverts commit c2e60ad0e5124482942164e5fec088157f5e716a.
    
    Turns out this check is excessively strict: there are ways
    other than <memtune><hard_limit> to raise the memory locking
    limit for QEMU processes, one prominent example being
    tweaking /etc/security/limits.conf.
    
    Partially-resolves: https://bugzilla.redhat.com/1431793

Comment 12 Luyao Huang 2017-06-02 02:51:52 UTC

Verify this bug with libvirt-3.2.0-7.el7.x86_64:

1. enable locked in memory backend without set hard limit:

# virsh dumpxml r7
<domain type='kvm' id='2'>
  <name>r7</name>
  <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid>
  <maxMemory slots='16' unit='KiB'>15243264</maxMemory>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>1024000</currentMemory>
  <memoryBacking>
    <locked/>
  </memoryBacking>


2. check qemu process memlock:

# prlimit -p `pidof qemu-kvm`
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0         0 blocks
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited blocks
LOCKS      max number of file locks held      unlimited unlimited 
MEMLOCK    max locked-in-memory address space unlimited unlimited bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                8192      8192 
NPROC      max number of processes               127505    127505 
RSS        max resident set size              unlimited unlimited pages
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals         127505    127505 
STACK      max stack size                       8388608 unlimited bytes


3. restart a guest with hard limit set in xml:

# virsh dumpxml r7
<domain type='kvm' id='4'>
  <name>r7</name>
  <uuid>67c7a123-5415-4136-af62-a2ee098ba6cd</uuid>
  <maxMemory slots='16' unit='KiB'>15243264</maxMemory>
  <memory unit='KiB'>1024000</memory>
  <currentMemory unit='KiB'>1024000</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>3024000</hard_limit>
  </memtune>
  <memoryBacking>
    <locked/>
  </memoryBacking>


# prlimit -p `pidof qemu-kvm`
RESOURCE   DESCRIPTION                              SOFT       HARD UNITS
AS         address space limit                 unlimited  unlimited bytes
CORE       max core file size                          0          0 blocks
CPU        CPU time                            unlimited  unlimited seconds
DATA       max data size                       unlimited  unlimited bytes
FSIZE      max file size                       unlimited  unlimited blocks
LOCKS      max number of file locks held       unlimited  unlimited 
MEMLOCK    max locked-in-memory address space 3096576000 3096576000 bytes
MSGQUEUE   max bytes in POSIX mqueues             819200     819200 bytes
NICE       max nice prio allowed to raise              0          0 
NOFILE     max number of open files                 8192       8192 
NPROC      max number of processes                127505     127505 
RSS        max resident set size               unlimited  unlimited pages
RTPRIO     max real-time priority                      0          0 
RTTIME     timeout for real-time tasks         unlimited  unlimited microsecs
SIGPENDING max number of pending signals          127505     127505 
STACK      max stack size                        8388608  unlimited bytes

Comment 13 errata-xmlrpc 2017-08-01 17:24:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 14 errata-xmlrpc 2017-08-02 00:03:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846