Bug 515692

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Can we get a QA ack for this one...

Links to upstream patches

http://www.redhat.com/archives/libvir-list/2010-October/msg00208.html

virDomain{Get,Set}MemoryParameters APIs were included in libvirt-0.8.5, moving to modified.

1. Without flags the current settings are correct displayed:
# virsh memtune vr-rhel6-x86_64-kvm
error: Unable to get memory parameters
error: internal error cannot find cgroup for domain vr-rhel6-x86_64-kvm

# virsh start vr-rhel6-x86_64-kvm
Domain vr-rhel6-x86_64-kvm started

# virsh memtune vr-rhel6-x86_64-kvm
hard_limit     : unlimited
soft_limit     : unlimited
swap_hard_limit: unlimited

and can get these values via cgget:

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.limit_in_bytes=9223372036854775807

# cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.soft_limit_in_bytes=9223372036854775807

# cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.memsw.limit_in_bytes=9223372036854775807

2. change value of --hard-limit, --soft-limit and --swap-hard-limit 
# virsh memtune vr-rhel6-x86_64-kvm --hard-limit 262144
error: Unable to change memory parameters
error: unable to set memory hard_limit tunable: Device or resource busy

Is this a normal behavior?

# virsh memtune vr-rhel6-x86_64-kvm --soft-limit 131072

I think libvirt should give a friendly output information in here.

# virsh memtune vr-rhel6-x86_64-kvm
hard_limit     : unlimited
soft_limit     : 131072 kB
swap_hard_limit: unlimited

# virsh memtune vr-rhel6-x86_64-kvm --swap-hard-limit 65536
error: Unable to change memory parameters
error: unable to set swap_hard_limit tunable: Invalid argument

I can't change value of --swap-hard-limit, I'm not sure whether my 
host machine need some extra configurations in here.

3. set current memory value to equal to value of --soft-limit:
# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1020.0s
Max memory:     1048576 kB
Used memory:    1048576 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)


# virsh setmem vr-rhel6-x86_64-kvm --kilobytes 131072

Same as above.

# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1023.4s
Max memory:     1048576 kB
Used memory:    160012 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)

What is the relationship between 160012 kB(Used memory) and 131072 kB? 

less than value of --soft-limit:

# virsh setmem vr-rhel6-x86_64-kvm --kilobytes 65536

Same as above.


# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1142.2s
Max memory:     1048576 kB
Used memory:    160012 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)

--soft-limit=131072kB, 65536 < 131072, but libvirt hasn't any error
information,
Is it right?

greater than value of --soft-limit:

# virsh setmem vr-rhel6-x86_64-kvm --kilobytes 262144

same as above.

# virsh dominfo vr-rhel6-x86_64-kvm
Id:             4
Name:           vr-rhel6-x86_64-kvm
UUID:           1dcc3c7a-4c6b-14cc-f434-603111ced616
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       1167.7s
Max memory:     1048576 kB
Used memory:    160012 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c236,c301 (permissive)

retry change values of --hard-limit and --swap-hard-limit :
# virsh memtune vr-rhel6-x86_64-kvm --hard-limit 262144

# virsh memtune vr-rhel6-x86_64-kvm
hard_limit     : 262144 kB
soft_limit     : 131072 kB
swap_hard_limit: unlimited

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6-x86_64-kvm
memory.limit_in_bytes=268435456

# virsh memtune vr-rhel6-x86_64-kvm --swap-hard-limit 65536
error: Unable to change memory parameters
error: unable to set swap_hard_limit tunable: Invalid argument


if we don't consider the above issues, are enough these testing for the bug?

Version-Release number of selected component (if applicable):

# rpm -q libvirt 
libvirt-0.8.7-7.el6.x86_64

# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.145.el6.x86_64

# uname -r
2.6.32-113.el6.x86_64

In addition, if directly add <memtune> xml block in the guest xml configuration,
it will be okay:

# virsh edit vr-rhel6u1-x86_64-kvm

# virsh dumpxml vr-rhel6u1-x86_64-kvm
......
  <memory>2097152</memory>
  <currentMemory>1048576</currentMemory>
  <memtune>
    <hard_limit>512000</hard_limit>
    <soft_limit>128000</soft_limit>
    <swap_hard_limit>1024000</swap_hard_limit>
  </memtune>
  <vcpu>1</vcpu>
......

# virsh start vr-rhel6u1-x86_64-kvm
Domain vr-rhel6u1-x86_64-kvm started

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : 128000 kB
swap_hard_limit: 1024000 kB

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.limit_in_bytes=524288000

# cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.soft_limit_in_bytes=131072000

# cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.memsw.limit_in_bytes=1048576000


For virsh command operation, I don't know why we can't to change value of swap_hard_limit, but directly update guest xml is okay, see also bug 652561.


Another question, when I set current memory size to 131072, the guest kernel will panic, and start guest again, sometimes the guest will hang, see also bug 652517.

# virsh setmem vr-rhel6u1-x86_64-kvm --kilobytes 131072

# virt-viewer vr-rhel6u1-x86_64-kvm

(In reply to comment #11)
> In addition, if directly add <memtune> xml block in the guest xml
> configuration,
> it will be okay:
> 
> # virsh edit vr-rhel6u1-x86_64-kvm
> 
> # virsh dumpxml vr-rhel6u1-x86_64-kvm
> ......
>   <memory>2097152</memory>
>   <currentMemory>1048576</currentMemory>
>   <memtune>
>     <hard_limit>512000</hard_limit>
>     <soft_limit>128000</soft_limit>
>     <swap_hard_limit>1024000</swap_hard_limit>
>   </memtune>
>   <vcpu>1</vcpu>
> ......
> 
> # virsh start vr-rhel6u1-x86_64-kvm
> Domain vr-rhel6u1-x86_64-kvm started
> 
> # virsh memtune vr-rhel6u1-x86_64-kvm
> hard_limit     : 512000 kB
> soft_limit     : 128000 kB
> swap_hard_limit: 1024000 kB
> 
> # cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
> memory.limit_in_bytes=524288000
> 
> # cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
> memory.soft_limit_in_bytes=131072000
> 
> # cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
> memory.memsw.limit_in_bytes=1048576000
> 
> 
> For virsh command operation, I don't know why we can't to change value of
> swap_hard_limit, but directly update guest xml is okay, see also bug 652561.
> 
The bug is 652561
> 
> Another question, when I set current memory size to 131072, the guest kernel
> will panic, and start guest again, sometimes the guest will hang, see also bug
> 652517.
The bug is 652517
> 
> # virsh setmem vr-rhel6u1-x86_64-kvm --kilobytes 131072
> 
> # virt-viewer vr-rhel6u1-x86_64-kvm

Reply Comment 12:
these 2 bugs are 680198 and 680190.

Retry to verify the feature:
# virsh dominfo vr-rhel6u1-x86_64-kvm
Id:             9
Name:           vr-rhel6u1-x86_64-kvm
UUID:           a03093a1-5da6-a2a2-3baf-a845db2f10b9
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       16.2s
Max memory:     2097152 kB
Used memory:    1024000 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c612,c921 (permissive)

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : unlimited
soft_limit     : unlimited
swap_hard_limit: unlimited

# virsh memtune vr-rhel6u1-x86_64-kvm --hard-limit 512000

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : unlimited
swap_hard_limit: unlimited

# cgget -nr memory.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.limit_in_bytes=524288000


# virsh memtune vr-rhel6u1-x86_64-kvm --soft-limit 128000

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : 128000 kB
swap_hard_limit: unlimited

# cgget -nr memory.soft_limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.soft_limit_in_bytes=131072000


# virsh memtune vr-rhel6u1-x86_64-kvm --swap-hard-limit 1024000

# virsh memtune vr-rhel6u1-x86_64-kvm
hard_limit     : 512000 kB
soft_limit     : 128000 kB
swap_hard_limit: 1024000 kB

# cgget -nr memory.memsw.limit_in_bytes libvirt/qemu/vr-rhel6u1-x86_64-kvm
memory.memsw.limit_in_bytes=1048576000

And also can directly edit guest xml configuration (see Comment 11) to get the above test result.


Some issues:

hard_limit is 512000 kB, it is maximum memory the guest domain can use.
However, I can set current memory to 1025000 kB > 512000 kB, it seems
hard_limit has no effect.

# virsh dominfo vr-rhel6u1-x86_64-kvm
Id:             9
Name:           vr-rhel6u1-x86_64-kvm
UUID:           a03093a1-5da6-a2a2-3baf-a845db2f10b9
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       20.6s
Max memory:     2097152 kB
Used memory:    1024000 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c612,c921 (permissive)

# virsh setmem vr-rhel6u1-x86_64-kvm --kilobytes 1025000

# virsh dominfo vr-rhel6u1-x86_64-kvm
Id:             9
Name:           vr-rhel6u1-x86_64-kvm
UUID:           a03093a1-5da6-a2a2-3baf-a845db2f10b9
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       21.6s
Max memory:     2097152 kB
Used memory:    1025000 kB
Persistent:     yes
Autostart:      disable
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c612,c921 (permissive)

Hi, Alex,

(In reply to comment #18)
> Hi, Alex,

Oops, sent a empty comment, :-)

For the problem you can't set "memsw.hard_limit" to "65536", it's duplicate problem with https://bugzilla.redhat.com/show_bug.cgi?id=680190, and it's already fixed upstream.

Bug 680198 cannot be fixed since we can't prevent users from shooting in their own legs without also preventing someone else from shooting a bug sitting on their leg :-)

Bug 680190 is a documentation bug.

And I filed a new bug 692468 for the last issue where virsh setmem allows to set more memory than what is allowed by virsh memtune. However, doing so doesn't actually allow the guest to use more memory. It's just that setmem and memtune are not integrated very well.

Setting the bug status to VERIFIED according to Comment 15 and Comment 20.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
    virDomainSetMemory() may not be honored by guest OS

Consequence
    it is not possible to set hard limits on memory used by a guest

Change
    new virDomain{Get,Set}MemoryParameters APIs were introduced which allow
    for fine-tuning various memory limits

Result
    memory limits for guests can be enforced

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,12 +1 @@
-Cause
+Guests were not required to honour the virDomainSetMemory() setting, making it impossible to set a hard limit on guest memory consumption. New virDomainGetMemoryParameters and virDomainSetMemoryParameters methods have been introduced to allow users to fine-tune and enforce memory limits.-    virDomainSetMemory() may not be honored by guest OS
-
-Consequence
-    it is not possible to set hard limits on memory used by a guest
-
-Change
-    new virDomain{Get,Set}MemoryParameters APIs were introduced which allow
-    for fine-tuning various memory limits
-
-Result
-    memory limits for guests can be enforced

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0596.html