RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 515692 - RFE: Use cgroups to enforce host RAM limit according to current memory balloon allocation
Summary: RFE: Use cgroups to enforce host RAM limit according to current memory balloo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.0
Hardware: All
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 519868 554544 575275 Rhel6.1LibvirtTier2 692468
TreeView+ depends on / blocked
 
Reported: 2009-08-05 12:23 UTC by Daniel Berrangé
Modified: 2015-09-28 02:31 UTC (History)
11 users (show)

Fixed In Version: libvirt-0.8.7-1.el6
Doc Type: Enhancement
Doc Text:
Guests were not required to honour the virDomainSetMemory() setting, making it impossible to set a hard limit on guest memory consumption. New virDomainGetMemoryParameters and virDomainSetMemoryParameters methods have been introduced to allow users to fine-tune and enforce memory limits.
Clone Of:
: 692468 (view as bug list)
Environment:
Last Closed: 2011-05-19 13:17:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0596 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2011-05-18 17:56:36 UTC

Description Daniel Berrangé 2009-08-05 12:23:34 UTC 
Description of problem:
The virDomainSetMemory() api call for the QEMU drivers uses the memory balloon device to request that the guest OS modify its allocation. The guest is not required to honour this setting. This could allow a mis-behaviing guest to force over commit of the host. CGroups has ability to set a memory limit on a group. By setting this to the same value as the memory balloon (+ some n MB overhead), we can prevent overcommit.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 2 RHEL Program Management 2009-08-05 12:50:04 UTC 
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 4 Hugh Brock 2009-12-21 17:03:45 UTC 
Can we get a QA ack for this one...
Comment 7 Daniel Berrangé 2010-10-08 15:23:25 UTC 
Links to upstream patches

http://www.redhat.com/archives/libvir-list/2010-October/msg00208.html
Comment 8 Jiri Denemark 2011-01-14 12:32:45 UTC 
virDomain{Get,Set}MemoryParameters APIs were included in libvirt-0.8.5, moving to modified.
Comment 10 Alex Jia 2011-02-23 02:16:06 UTC 
1. Without flags the current settings are correct displayed:
# virsh memtune vr-rhel6-x86_64-kvm
error: Unable to get memory parameters
error: internal error cannot find cgroup for domain vr-rhel6-x86_64-kvm

# virsh start vr-rhel6-x86_64-kvm
Domain vr-rhel6-x86_64-kvm started

# virsh memtune vr-rhel6-x86_64-kvm
hard_limit     : unlimited
soft_limit     : unlimited
swap_hard_limit: unlimited

and can get these values via cgget:

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.limit_in_bytes=9223372036854775807

# cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.soft_limit_in_bytes=9223372036854775807

# cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.memsw.limit_in_bytes=9223372036854775807

2. change value of --hard-limit, --soft-limit and --swap-hard-limit 
# virsh memtune vr-rhel6-x86_64-kvm --hard-limit 262144
error: Unable to change memory parameters
error: unable to set memory hard_limit tunable: Device or resource busy

Is this a normal behavior?

# virsh memtune vr-rhel6-x86_64-kvm --soft-limit 131072

I think libvirt should give a friendly output information in here.

# virsh memtune vr-rhel6-x86_64-kvm
hard_limit     : unlimited
soft_limit     : 131072 kB
swap_hard_limit: unlimited

# virsh memtune vr-rhel6-x86_64-kvm --swap-hard-limit 65536
error: Unable to change memory parameters
error: unable to set swap_hard_limit tunable: Invalid argument

I can't change value of --swap-hard-limit, I'm not sure whether my 
host machine need some extra configurations in here.

3. set current memory value to equal to value of --soft-limit:
# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1020.0s
Max memory:     1048576 kB
Used memory:    1048576 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)


# virsh setmem vr-rhel6-x86_64-kvm --kilobytes 131072

Same as above.

# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1023.4s
Max memory:     1048576 kB
Used memory:    160012 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)

What is the relationship between 160012 kB(Used memory) and 131072 kB? 

less than value of --soft-limit:

# virsh setmem vr-rhel6-x86_64-kvm --kilobytes 65536

Same as above.


# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1142.2s
Max memory:     1048576 kB
Used memory:    160012 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)

--soft-limit=131072kB, 65536 < 131072, but libvirt hasn't any error
information,
Is it right?

greater than value of --soft-limit:

# virsh setmem vr-rhel6-x86_64-kvm --kilobytes 262144

same as above.

# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1167.7s
Max memory:     1048576 kB
Used memory:    160012 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)

retry change values of --hard-limit and --swap-hard-limit :
# virsh memtune vr-rhel6-x86_64-kvm --hard-limit 262144

# virsh memtune vr-rhel6-x86_64-kvm
hard_limit     : 262144 kB
soft_limit     : 131072 kB
swap_hard_limit: unlimited

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.limit_in_bytes=268435456

# virsh memtune vr-rhel6-x86_64-kvm --swap-hard-limit 65536
error: Unable to change memory parameters
error: unable to set swap_hard_limit tunable: Invalid argument


if we don't consider the above issues, are enough these testing for the bug?

Version-Release number of selected component (if applicable):

# rpm -q libvirt 
libvirt-0.8.7-7.el6.x86_64

# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.145.el6.x86_64

# uname -r
2.6.32-113.el6.x86_64
Comment 11 Alex Jia 2011-02-23 04:52:53 UTC 
In addition, if directly add <memtune> xml block in the guest xml configuration,
it will be okay:

# virsh edit vr-rhel6u1-x86_64-kvm

# virsh dumpxml vr-rhel6u1-x86_64-kvm
......
  <memory>2097152</memory>
  <currentMemory>1048576</currentMemory>
  <memtune>
    <hard_limit>512000</hard_limit>
    <soft_limit>128000</soft_limit>
    <swap_hard_limit>1024000</swap_hard_limit>
  </memtune>
  <vcpu>1</vcpu>
......

# virsh start vr-rhel6u1-x86_64-kvm
Domain vr-rhel6u1-x86_64-kvm started

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : 128000 kB
swap_hard_limit: 1024000 kB

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.limit_in_bytes=524288000

# cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.soft_limit_in_bytes=131072000

# cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.memsw.limit_in_bytes=1048576000


For virsh command operation, I don't know why we can't to change value of swap_hard_limit, but directly update guest xml is okay, see also bug 652561.


Another question, when I set current memory size to 131072, the guest kernel will panic, and start guest again, sometimes the guest will hang, see also bug 652517.

# virsh setmem vr-rhel6u1-x86_64-kvm --kilobytes 131072

# virt-viewer vr-rhel6u1-x86_64-kvm
Comment 12 Alex Jia 2011-02-23 08:47:21 UTC 
(In reply to comment #11)
> In addition, if directly add <memtune> xml block in the guest xml
> configuration,
> it will be okay:
> 
> # virsh edit vr-rhel6u1-x86_64-kvm
> 
> # virsh dumpxml vr-rhel6u1-x86_64-kvm
> ......
>   <memory>2097152</memory>
>   <currentMemory>1048576</currentMemory>
>   <memtune>
>     <hard_limit>512000</hard_limit>
>     <soft_limit>128000</soft_limit>
>     <swap_hard_limit>1024000</swap_hard_limit>
>   </memtune>
>   <vcpu>1</vcpu>
> ......
> 
> # virsh start vr-rhel6u1-x86_64-kvm
> Domain vr-rhel6u1-x86_64-kvm started
> 
> # virsh memtune vr-rhel6u1-x86_64-kvm
> hard_limit     : 512000 kB
> soft_limit     : 128000 kB
> swap_hard_limit: 1024000 kB
> 
> # cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
> memory.limit_in_bytes=524288000
> 
> # cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
> memory.soft_limit_in_bytes=131072000
> 
> # cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
> memory.memsw.limit_in_bytes=1048576000
> 
> 
> For virsh command operation, I don't know why we can't to change value of
> swap_hard_limit, but directly update guest xml is okay, see also bug 652561.
> 
The bug is 652561
> 
> Another question, when I set current memory size to 131072, the guest kernel
> will panic, and start guest again, sometimes the guest will hang, see also bug
> 652517.
The bug is 652517
> 
> # virsh setmem vr-rhel6u1-x86_64-kvm --kilobytes 131072
> 
> # virt-viewer vr-rhel6u1-x86_64-kvm
Comment 14 Alex Jia 2011-02-24 16:26:09 UTC 
Reply Comment 12:
these 2 bugs are 680198 and 680190.
Comment 15 Alex Jia 2011-03-07 06:27:49 UTC 
Retry to verify the feature:
# virsh dominfo vr-rhel6u1-x86_64-kvm
Id:             9
Name:           vr-rhel6u1-x86_64-kvm
UUID:           a03093a1-5da6-a2a2-3baf-a845db2f10b9
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       16.2s
Max memory:     2097152 kB
Used memory:    1024000 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c612,c921 (permissive)

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : unlimited
soft_limit     : unlimited
swap_hard_limit: unlimited

# virsh memtune vr-rhel6u1-x86_64-kvm --hard-limit 512000

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : unlimited
swap_hard_limit: unlimited

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.limit_in_bytes=524288000


# virsh memtune vr-rhel6u1-x86_64-kvm --soft-limit 128000

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : 128000 kB
swap_hard_limit: unlimited

# cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.soft_limit_in_bytes=131072000


# virsh memtune vr-rhel6u1-x86_64-kvm --swap-hard-limit 1024000

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : 128000 kB
swap_hard_limit: 1024000 kB

# cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.memsw.limit_in_bytes=1048576000

And also can directly edit guest xml configuration (see Comment 11) to get the above test result.


Some issues:

hard_limit is 512000 kB, it is maximum memory the guest domain can use.
However, I can set current memory to 1025000 kB > 512000 kB, it seems
hard_limit has no effect.

# virsh dominfo vr-rhel6u1-x86_64-kvm
Id:             9
Name:           vr-rhel6u1-x86_64-kvm
UUID:           a03093a1-5da6-a2a2-3baf-a845db2f10b9
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       20.6s
Max memory:     2097152 kB
Used memory:    1024000 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c612,c921 (permissive)

# virsh setmem vr-rhel6u1-x86_64-kvm --kilobytes 1025000

# virsh dominfo vr-rhel6u1-x86_64-kvm
Id:             9
Name:           vr-rhel6u1-x86_64-kvm
UUID:           a03093a1-5da6-a2a2-3baf-a845db2f10b9
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       21.6s
Max memory:     2097152 kB
Used memory:    1025000 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c612,c921 (permissive)
Comment 18 Osier Yang 2011-03-31 02:33:16 UTC 
Hi, Alex,
Comment 19 Osier Yang 2011-03-31 09:42:43 UTC 
(In reply to comment #18)
> Hi, Alex,

Oops, sent a empty comment, :-)

For the problem you can't set "memsw.hard_limit" to "65536", it's duplicate problem with https://bugzilla.redhat.com/show_bug.cgi?id=680190, and it's already fixed upstream.
Comment 20 Jiri Denemark 2011-03-31 12:23:37 UTC 
Bug 680198 cannot be fixed since we can't prevent users from shooting in their own legs without also preventing someone else from shooting a bug sitting on their leg :-)

Bug 680190 is a documentation bug.

And I filed a new bug 692468 for the last issue where virsh setmem allows to set more memory than what is allowed by virsh memtune. However, doing so doesn't actually allow the guest to use more memory. It's just that setmem and memtune are not integrated very well.
Comment 22 Alex Jia 2011-04-01 02:22:00 UTC 
Setting the bug status to VERIFIED according to Comment 15 and Comment 20.
Comment 23 Jiri Denemark 2011-05-03 12:47:19 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
    virDomainSetMemory() may not be honored by guest OS

Consequence
    it is not possible to set hard limits on memory used by a guest

Change
    new virDomain{Get,Set}MemoryParameters APIs were introduced which allow
    for fine-tuning various memory limits

Result
    memory limits for guests can be enforced
Comment 26 Laura Bailey 2011-05-04 01:25:02 UTC 
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,12 +1 @@
-Cause
+Guests were not required to honour the virDomainSetMemory() setting, making it impossible to set a hard limit on guest memory consumption. New virDomainGetMemoryParameters and virDomainSetMemoryParameters methods have been introduced to allow users to fine-tune and enforce memory limits.-    virDomainSetMemory() may not be honored by guest OS
-
-Consequence
-    it is not possible to set hard limits on memory used by a guest
-
-Change
-    new virDomain{Get,Set}MemoryParameters APIs were introduced which allow
-    for fine-tuning various memory limits
-
-Result
-    memory limits for guests can be enforced
Comment 27 errata-xmlrpc 2011-05-19 13:17:59 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0596.html
Note You need to log in before you can comment on or make changes to this bug.