Bug 829241

Summary: In LXC two same setmem commands get different results (for minor value)
Product: Red Hat Enterprise Linux 6 Reporter: EricLee <bili>
Component: libvirtAssignee: Martin Kletzander <mkletzan>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: acathrow, cwei, dallan, dyuan, jdenemar, lsu, mzhan, shyu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 908254 (view as bug list) Environment:
Last Closed: 2014-04-04 22:56:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 908254    
Bug Blocks:    
Attachments:
Description Flags
libvirtd.log
none
toy.log none

Description EricLee 2012-06-06 09:55:57 UTC
Description of problem:
In LXC two same setmem commands get different results (for minor value)

Version-Release number of selected component (if applicable):
# rpm -qa libvirt qemu-kvm kernel
qemu-kvm-0.12.1.2-2.295.el6.x86_64
kernel-2.6.32-276.el6.x86_64
libvirt-0.9.10-21.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1.# virsh -c lxc:/// list --all
 Id    Name                           State
----------------------------------------------------
 14620 toy                            running

2.#virsh # dumpxml toy
<domain type='lxc'>
  <name>toy</name>
  <uuid>386f5b25-43ee-9d62-4ce2-62c3809e47c1</uuid>
  <memory unit='KiB'>500000</memory>
  <currentMemory unit='KiB'>412300</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/bin/sh</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <interface type='network'>
      <mac address='52:54:00:82:92:e6'/>
      <source network='default'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

3.#virsh # dominfo toy
Id:             14620
Name:           toy
UUID:           386f5b25-43ee-9d62-4ce2-62c3809e47c1
OS Type:        exe
State:          running
CPU(s):         1
CPU time:       0.0s
Max memory:     500000 kB
Used memory:    536 kB
Persistent:     yes
Autostart:      disable
Managed save:   unknown
Security model: selinux
Security DOI:   0
Security label: unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 (enforcing)

4.virsh # setmem toy 300
error: operation failed: Failed to set memory for domain

5.virsh # setmem toy 300

6.virsh # dominfo toy
Id:             14620
Name:           toy
UUID:           386f5b25-43ee-9d62-4ce2-62c3809e47c1
OS Type:        exe
State:          running
CPU(s):         1
CPU time:       0.0s
Max memory:     500000 kB
Used memory:    176 kB
Persistent:     yes
Autostart:      disable
Managed save:   unknown
Security model: selinux
Security DOI:   0
Security label: unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 (enforcing)

Two same setmem command get different results.
  
Actual results:
As above

Expected results:
For step 3, should not get error, and get same result as step 4 and step 5.

Additional info:
For superior value will not get error.

Comment 1 EricLee 2012-06-06 09:58:44 UTC
(In reply to comment #0)
........
> 
> Expected results:
> For step 3, should not get error, and get same result as step 4 and step 5.
> 
Sorry,  Expected results  should be " For step 4, should not get error, and get same result as step 5 and step 6."

> Additional info:
> For superior value will not get error.

Comment 2 Martin Kletzander 2012-06-08 07:43:14 UTC
I have to confirm this, but most probably this is a dup of Bug #767921. Could you run libvirtd with LIBVIRT_DEBUG=1, reproduce it and attach a /var/log/libvirt/lxc/toy.log when you have some spare time? Thanks

Comment 3 EricLee 2012-06-08 08:00:02 UTC
Please see next two comments:
1.libvitd.log for LIBVIRT_DEBUG=1;
2.toy.log.

Comment 4 EricLee 2012-06-08 08:01:01 UTC
Created attachment 590371 [details]
libvirtd.log

Comment 5 EricLee 2012-06-08 08:01:32 UTC
Created attachment 590372 [details]
toy.log

Comment 6 Martin Kletzander 2012-07-02 10:16:23 UTC
This is the same problem as with Bug #767921, for which there will be a workaround, however still this is a duplicate.

*** This bug has been marked as a duplicate of bug 767921 ***

Comment 8 EricLee 2013-01-22 05:57:04 UTC
Hi Martin,

I can still reproduce this bug with the latest package:
libvirt-0.10.2-16.el6.x86_64

# virsh -c lxc:/// define toy.xml 
Domain toy defined from toy.xml

# virsh -c lxc:/// start toy
Domain toy started

# virsh -c lxc:/// list 
 Id    Name                           State
----------------------------------------------------
 15353 toy                            running

# virsh -c lxc:///
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list --all
 Id    Name                           State
----------------------------------------------------
 15353 toy                            running

virsh # dominfo toy
Id:             15353
Name:           toy
UUID:           386f5b25-43ee-9d62-4ce2-62c3809e47c1
OS Type:        exe
State:          running
CPU(s):         1
CPU time:       0.0s
Max memory:     500000 KiB
Used memory:    476 KiB
Persistent:     yes
Autostart:      disable
Managed save:   unknown

virsh # setmem toy 300
error: operation failed: Failed to set memory for domain

----> the first time to run this command will get error.

virsh # setmem toy 300

----> the second time do not get error.

virsh # dominfo toy
Id:             15353
Name:           toy
UUID:           386f5b25-43ee-9d62-4ce2-62c3809e47c1
OS Type:        exe
State:          running
CPU(s):         1
CPU time:       0.0s
Max memory:     500000 KiB
Used memory:    252 KiB
Persistent:     yes
Autostart:      disable
Managed save:   unknown

Was this a wrong closing as the Bug 767921(which this bug duplicated to) closed as duplicate to Bug 767921, and Bug 767921 has been verified?

Or this issue is caused by Bug 854552 - Cgroup out of memory ?

Should I re-open this bug or file another one ?

Thanks,
EricLee

Comment 9 Martin Kletzander 2013-01-22 16:27:56 UTC
The situation you're describing is reproducible without libvirt and everything comes from the kernel.  But I wouldn't necessarily say it is a bug.  When I check the memory usage, it is more than 300KiB before the limitation.  Limiting to 300KiB doesn't succeed, because kernel doesn't swap it out quickly enough.  The second try it succeeds, because enough of the data is swapped out already.

What is weird, though, is that I'm not experiencing this with kernel version 3.7.3 and process with bigger memory footprint, even though the neither the mem_cgroup_resize_limit() code nor MEM_CGROUP_RECLAIM_RETRIES has changed significantly between these two versions.  But I'm not a kernel developer, so that statement has most probably zero value.

I'd try reopening this on kernel to see if there is a possibility to fix it in an easy way.  I see it's not a dup of 767921 now, so sorry for that.

Comment 10 Luwen Su 2013-02-06 09:25:40 UTC
Per comment 9 , i reopen this bug to 6.5 with TestOnly flag only for tracking

Comment 12 Luwen Su 2013-10-30 06:26:23 UTC
Test again with
libvirt-0.10.2-29.el6.x86_64
kernel-2.6.32-426.el6.x86_64

virsh # dominfo test
Id:             2797
Name:           test
UUID:           2241b339-99e2-8224-3b10-102b5d9b689f
OS Type:        exe
State:          running
CPU(s):         1
CPU time:       0.0s
Max memory:     1048576 KiB
Used memory:    684 KiB
Persistent:     yes
Autostart:      disable
Managed save:   unknown

virsh # setmem test 300
error: operation failed: Failed to set memory for domain

virsh # setmem test 300
error: operation failed: Failed to set memory for domain

virsh # setmem test 300

virsh # dominfo test
Id:             2797
Name:           test
UUID:           2241b339-99e2-8224-3b10-102b5d9b689f
OS Type:        exe
State:          running
CPU(s):         1
CPU time:       0.0s
Max memory:     1048576 KiB
Used memory:    284 KiB
Persistent:     yes
Autostart:      disable
Managed save:   unknown


So does it need move to 6.6 or change back to assign ?

Comment 13 Jiri Denemark 2013-11-01 09:58:37 UTC
I think comment 9 still applies. Have you tried to open a bug for kernel as Martin suggested there?

Comment 14 Luwen Su 2013-11-01 10:16:40 UTC
Yes , see the depends on bug 908254

Comment 15 Jiri Denemark 2013-11-01 10:54:59 UTC
Ah, I'm apparently blind. I don't think we can do much without changing kernel behavior. Hmm, actually, I guess we could make it behave the same way it works in qemu driver where setmem is just a polite request to shrink the memory. If a specific errno is returned by kernel when shrinking memory doesn't succeed immediately because swapping takes some time, we could detect that error condition and ignore it. This way, setmem would succeed even though the memory was not shrunk to the requested size. In case this is not a viable solution, I suggest closing as CANTFIX. In any case, let's move this bug to 6.6 for further investigation.

Comment 19 RHEL Program Management 2014-04-04 22:56:14 UTC
Development Management has reviewed and declined this request.
You may appeal this decision by reopening this request.