Bug 839537 - Error occurs when given hard_limit in memtune more than current swap_hard_limit
Error occurs when given hard_limit in memtune more than current swap_hard_limit
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.4
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Osier Yang
Virtualization Bugs
:
: 767921 (view as bug list)
Depends On:
Blocks: 826880
  Show dependency treegraph
 
Reported: 2012-07-12 04:57 EDT by hongming
Modified: 2013-02-21 02:19 EST (History)
9 users (show)

See Also:
Fixed In Version: libvirt-0.10.1-1.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-21 02:19:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
memtune fail (64.43 KB, application/octet-stream)
2012-08-24 00:37 EDT, Luwen Su
no flags Details
kernel-log (8.27 KB, text/plain)
2012-09-03 06:52 EDT, Luwen Su
no flags Details

  None (edit)
Description hongming 2012-07-12 04:57:55 EDT
Description of problem:


Version-Release number of selected component (if applicable):
kernel- 2.6.32-276.el6.x86_64
libvirt-0.9.13-2.el6.x86_64
qemu-kvm-0.12.1.2-2.295.el6.x86_64

How reproducible:


Steps to Reproduce:
1.# virsh memtune rhel6 
hard_limit     : 1000020
soft_limit     : 1000020
swap_hard_limit: 1000020


2.# virsh memtune rhel6 --hard-limit 1000030 --soft-limit 1000020 --swap-hard-limit 1000040 --live  ( If --hard-limit 1000030 > swap_hard_limit in Step 1 , it failed)
error: Unable to change memory parameters
error: unable to set memory hard_limit tunable: Invalid argument


3.# virsh memtune rhel6 --hard-limit 1000020 --soft-limit 1000020 --swap-hard-limit 1000040 --live ( If --hard-limit 1000020 <= swap_hard_limit in Step 1 , it succeed)

4.# # virsh memtune rhel6
hard_limit     : 1000020
soft_limit     : 1000020
swap_hard_limit: 1000040

  
Actual results:

Error occurs when given  hard_limit in memtune more than current swap_hard_limit .

error: Unable to change memory parameters
error: unable to set memory hard_limit tunable: Invalid argument

Expected results:
The given hard_limit should only compare with the given swap-hard-limit in command.

Additional info:
Comment 2 Osier Yang 2012-07-12 05:08:41 EDT
Patch posted upstream:

https://www.redhat.com/archives/libvir-list/2012-July/msg00535.html
Comment 3 Dave Allan 2012-07-23 09:42:56 EDT
*** Bug 767921 has been marked as a duplicate of this bug. ***
Comment 4 Osier Yang 2012-08-17 10:09:41 EDT
Commit 75e5bec97 fixed the problem.
Comment 6 Luwen Su 2012-08-24 00:34:12 EDT
Seems there are  some problems.So , accroding my steps , should we open a new bz or keep this one to track the problem?

------------------------------------------------------------------
first for verfy the bug:
# virsh start test-1
Domain test-1 started
# virsh memtune test-1
hard_limit     : 1283748
soft_limit     : unlimited
swap_hard_limit: unlimited

# virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020 --swap-hard-limit 1000020 --live 

# virsh memtune test-1
hard_limit     : 1000020
soft_limit     : 1000020
swap_hard_limit: 1000020

# virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020 --swap-hard-limit 1000040 --live 

# virsh memtune test-1
hard_limit     : 1000032
soft_limit     : 1000020
swap_hard_limit: 1000040

# virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020 --swap-hard-limit 1000040 --live 
error: Unable to change memory parameters
error: Requested operation is not valid: domain is not running

[root@localhost /]# virsh memtune test-1
hard_limit     : 0
soft_limit     : 0
swap_hard_limit: 0

------------------------------------------------------------------------
Below are other situations i test.


1.
a shutoff guest
# virsh memtune test-1
hard_limit     : 1283748
soft_limit     : unlimited
swap_hard_limit: unlimited

# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020
error: Unable to change memory parameters
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor

# service libvirtd status
libvirtd dead but pid file exists

libvirtd.log:see the attachment 


2
#service libvirtd restart
#virsh start test-1
# virsh memtune test-1
hard_limit     : 1283748
soft_limit     : unlimited
swap_hard_limit: unlimited
# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020
the guest is atuo be destroyed 
libvirtd.log:
2012-08-24 04:20:03.076+0000: 6090: info : libvirt version: 0.10.0, package: 0rc1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-08-23-00:04:57, x86-007.build.bos.redhat.com)
2012-08-24 04:20:03.076+0000: 6090: error : virDBusGetSystemBus:193 : internal error DBus support not compiled into this binary
2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 3
2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 4
2012-08-24 04:20:09.029+0000: 6080: warning : qemuSetupCgroupForEmulator:659 : cgroup 5 is not active
2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 6
2012-08-24 04:20:26.062+0000: 6079: error : qemuMonitorIO:602 : internal error End of file from monitor


3.
# virsh start test-1
Domain test-1 started
# virsh memtune test-1
hard_limit     : 1283748
soft_limit     : unlimited
swap_hard_limit: unlimited

# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100021 --live

# virsh memtune test-1
hard_limit     : 100000
soft_limit     : 100000
swap_hard_limit: 100024

libvirtd.log:
2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 3
2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 4
2012-08-24 04:21:56.626+0000: 6080: warning : qemuSetupCgroupForEmulator:659 : cgroup 5 is not active
2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 6

4.
virsh start test-1
Domain test-1 started

# virsh memtune test-1
hard_limit     : 1283748
soft_limit     : unlimited
swap_hard_limit: unlimited

# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100025 --live

# virsh memtune test-1
hard_limit     : 100000
soft_limit     : 100000
swap_hard_limit: 100028     <----different with 25

# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100026 --live

# virsh memtune test-1
hard_limit     : 100000
soft_limit     : 100000
swap_hard_limit: 100028

# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100029 --live

# virsh memtune test-1
hard_limit     : 100000
soft_limit     : 100000
swap_hard_limit: 100032

# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100031 --live
error: Unable to change memory parameters
error: Requested operation is not valid: domain is not running

# virsh memtune test-1
hard_limit     : 0
soft_limit     : 0
swap_hard_limit: 0
Comment 7 Luwen Su 2012-08-24 00:37:05 EDT
Created attachment 606749 [details]
memtune fail
Comment 8 Osier Yang 2012-08-27 22:21:52 EDT
(In reply to comment #6)
> Seems there are  some problems.So , accroding my steps , should we open a
> new bz or keep this one to track the problem?
> 
> ------------------------------------------------------------------
> first for verfy the bug:
> # virsh start test-1
> Domain test-1 started
> # virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : unlimited
> swap_hard_limit: unlimited
> 
> # virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020
> --swap-hard-limit 1000020 --live 
> 
> # virsh memtune test-1
> hard_limit     : 1000020
> soft_limit     : 1000020
> swap_hard_limit: 1000020
> 
> # virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020
> --swap-hard-limit 1000040 --live 
> 
> # virsh memtune test-1
> hard_limit     : 1000032
> soft_limit     : 1000020
> swap_hard_limit: 1000040
> 
> # virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020
> --swap-hard-limit 1000040 --live 
> error: Unable to change memory parameters
> error: Requested operation is not valid: domain is not running

So what's the domain state at this point? could it be shutdown by another
thread?

> 
> [root@localhost /]# virsh memtune test-1
> hard_limit     : 0
> soft_limit     : 0
> swap_hard_limit: 0
> 
> ------------------------------------------------------------------------
> Below are other situations i test.
> 
> 
> 1.
> a shutoff guest
> # virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : unlimited
> swap_hard_limit: unlimited
> 
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100020
> error: Unable to change memory parameters
> error: End of file while reading data: Input/output error
> error: Failed to reconnect to the hypervisor

Could you attach the gdb backstrace?

> 
> # service libvirtd status
> libvirtd dead but pid file exists
> 
> libvirtd.log:see the attachment 
> 
> 
> 2 [details]
> #service libvirtd restart
> #virsh start test-1
> # virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : unlimited
> swap_hard_limit: unlimited
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100020
> the guest is atuo be destroyed 
> libvirtd.log:
> 2012-08-24 04:20:03.076+0000: 6090: info : libvirt version: 0.10.0, package:
> 0rc1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
> 2012-08-23-00:04:57, x86-007.build.bos.redhat.com)
> 2012-08-24 04:20:03.076+0000: 6090: error : virDBusGetSystemBus:193 :
> internal error DBus support not compiled into this binary
> 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm
> cgroup in controller 3
> 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm
> cgroup in controller 4
> 2012-08-24 04:20:09.029+0000: 6080: warning : qemuSetupCgroupForEmulator:659
> : cgroup 5 is not active
> 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm
> cgroup in controller 6
> 2012-08-24 04:20:26.062+0000: 6079: error : qemuMonitorIO:602 : internal
> error End of file from monitor

This tells the domain is shutdown. I'm wondering what causes that.

> 
> 
> 3.
> # virsh start test-1
> Domain test-1 started
> # virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : unlimited
> swap_hard_limit: unlimited
> 
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100021 --live
> 
> # virsh memtune test-1
> hard_limit     : 100000
> soft_limit     : 100000
> swap_hard_limit: 100024
> 
> libvirtd.log:
> 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm
> cgroup in controller 3
> 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm
> cgroup in controller 4
> 2012-08-24 04:21:56.626+0000: 6080: warning : qemuSetupCgroupForEmulator:659
> : cgroup 5 is not active
> 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm
> cgroup in controller 6
> 
> 4.
> virsh start test-1
> Domain test-1 started
> 
> # virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : unlimited
> swap_hard_limit: unlimited
> 
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100025 --live
> 
> # virsh memtune test-1
> hard_limit     : 100000
> soft_limit     : 100000
> swap_hard_limit: 100028     <----different with 25

Fine, should be rounded up by cgroup.

> 
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100026 --live
> 
> # virsh memtune test-1
> hard_limit     : 100000
> soft_limit     : 100000
> swap_hard_limit: 100028
> 
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100029 --live
> 
> # virsh memtune test-1
> hard_limit     : 100000
> soft_limit     : 100000
> swap_hard_limit: 100032
> 
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100031 --live
> error: Unable to change memory parameters
> error: Requested operation is not valid: domain is not running
> 
> # virsh memtune test-1
> hard_limit     : 0
> soft_limit     : 0
> swap_hard_limit: 0
Comment 9 Luwen Su 2012-08-27 23:19:18 EDT
(In reply to comment #8)
> (In reply to comment #6)
> > Seems there are  some problems.So , accroding my steps , should we open a
> > new bz or keep this one to track the problem?

I test the the situations above again

1.All the behavior cause libvirtd crashed have the same gdb information:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff06b65c700 (LWP 19506)]
0x0000003de484a257 in virCgroupPathOfController () from /usr/lib64/libvirt.so.0
(gdb) backtrace
#0  0x0000003de484a257 in virCgroupPathOfController () from /usr/lib64/libvirt.so.0
#1  0x0000003de484a733 in ?? () from /usr/lib64/libvirt.so.0
#2  0x0000003de484b12c in ?? () from /usr/lib64/libvirt.so.0
#3  0x0000003de484b1ee in virCgroupGetMemSwapHardLimit () from /usr/lib64/libvirt.so.0
#4  0x00000000004646bd in ?? ()
#5  0x0000003de48f6640 in virDomainSetMemoryParameters () from /usr/lib64/libvirt.so.0
#6  0x000000000043f079 in ?? ()
#7  0x0000003de492ee3d in virNetServerProgramDispatch () from /usr/lib64/libvirt.so.0
#8  0x0000003de492fade in ?? () from /usr/lib64/libvirt.so.0
#9  0x0000003de493017c in ?? () from /usr/lib64/libvirt.so.0
#10 0x0000003de485f30c in ?? () from /usr/lib64/libvirt.so.0
#11 0x0000003de485ebf9 in ?? () from /usr/lib64/libvirt.so.0
#12 0x0000003dcd407851 in start_thread () from /lib64/libpthread.so.0
#13 0x0000003dcd0e767d in clone () from /lib64/libc.so.6


2.For the Verify steps , it works fine and i can't reproduce it now . I still research what the reason cause  failure.
Comment 10 Osier Yang 2012-08-30 00:37:19 EDT
commit 20b780aec9d48888c5088b97887d268df77884c0
Author: Osier Yang <jyang@redhat.com>
Date:   Wed Aug 29 14:46:06 2012 +0800

    qemu: Sort the numa params only when it affects the live config
    
    As the next boot doesn't have to worry about the previous numa
    params setting (there is no).

A follow up patch to fix the crash.
Comment 11 Luwen Su 2012-09-03 03:44:36 EDT
Test with libvirt-0.10.1.1 again.
Good news is libvirt don't cras.
But it seems still have some issues.So change to assign.

libvirt-0.10.1-1.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.307.el6.x86_64
kernel-2.6.32-298.el6.x86_64

1.
 virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020
 --swap-hard-limit 1000040 --live 
 
 # virsh memtune test-1
 hard_limit     : 1000032     <----the number doesn't accurate yet
 soft_limit     : 1000020
 swap_hard_limit: 1000040

the libvirtd.log's warning still exist 
warning : virCgroupMoveTask:885 : no vm cgroup in controller 3

2.
a shutoff guest
# virsh memtune test-1
hard_limit     : 1283748
soft_limit     : unlimited
swap_hard_limit: unlimited
# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020

It works , but qemu hang this time then libvirt get a 
Timed out during operation: cannot acquire state change lock

libvirtd.log (omit the time info):
warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process 18194, sending SIGKILL
 warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process 18194
 info : libvirt version: 0.10.1, package: 1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-08-31-10:30:29, x86-009.build.bos.redhat.com)

error : virNetSocketReadWire:1184 : End of file while reading data: Input/output error
warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process 18194, sending SIGKILL
 warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process 18194
 warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process 18194, sending SIGKILL
warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process 18194
warning : qemuDomainObjBeginJobInternal:838 : Cannot start job (destroy, none) for domain test; current job is (modify, none) owned by (18474, 0)
 error : qemuDomainObjBeginJobInternal:842 : Timed out during operation: cannot acquire state change lock
Comment 12 Osier Yang 2012-09-03 05:47:42 EDT
(In reply to comment #11)
> Test with libvirt-0.10.1.1 again.
> Good news is libvirt don't cras.
> But it seems still have some issues.So change to assign.
> 
> libvirt-0.10.1-1.el6.x86_64
> qemu-kvm-rhev-0.12.1.2-2.307.el6.x86_64
> kernel-2.6.32-298.el6.x86_64
> 
> 1.
>  virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020
>  --swap-hard-limit 1000040 --live 
>  
>  # virsh memtune test-1
>  hard_limit     : 1000032     <----the number doesn't accurate yet

See my comment in comment 8, you will known that if you tested with cgroup
directly.

>  soft_limit     : 1000020
>  swap_hard_limit: 1000040
> 
> the libvirtd.log's warning still exist 
> warning : virCgroupMoveTask:885 : no vm cgroup in controller 3

A warning can't be a bug, unless there is something crazy there. And the warning
is  not caused by setting cgroup for emulartor, it's not from memory tuning cgroup. So if you think the warning has some problems, file a separate bug.

> 
> 2.
> a shutoff guest
> # virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : unlimited
> swap_hard_limit: unlimited
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100020
> 
> It works , but qemu hang this time then libvirt get a 
> Timed out during operation: cannot acquire state change lock

I don't understand it here. Mainly questions are:
1) what does "it works" mean here, does it mean 'memtune' return successfully?
   or it hangs there?

2) what you did right before and after "memtune". As the error "timeout to
   acquire the lock" is about killing the domain process. I don't think it
   occurs when doing "memtune", as you declared the domain was already in
   shutoff state before executing "memtune" command.

3) I don't see any dead lock possbility in "memtune" API. It's very likely
   you did something else, that's why what I asked "what you did right before
   and after memtune command".

> 
> libvirtd.log (omit the time info):
> warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process
> 18194, sending SIGKILL
>  warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process
> 18194
>  info : libvirt version: 0.10.1, package: 1.el6 (Red Hat, Inc.
> <http://bugzilla.redhat.com/bugzilla>, 2012-08-31-10:30:29,
> x86-009.build.bos.redhat.com)
> 
> error : virNetSocketReadWire:1184 : End of file while reading data:
> Input/output error
> warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process
> 18194, sending SIGKILL
>  warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process
> 18194
>  warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process
> 18194, sending SIGKILL
> warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process
> 18194
> warning : qemuDomainObjBeginJobInternal:838 : Cannot start job (destroy,
> none) for domain test; current job is (modify, none) owned by (18474, 0)
>  error : qemuDomainObjBeginJobInternal:842 : Timed out during operation:
> cannot acquire state change lock
Comment 13 Luwen Su 2012-09-03 06:50:32 EDT
(In reply to comment #12)
> (In reply to comment #11)
I'm sorry about my comments make you confuse.You're right , it's fine for a shutdown guest that do any memtune actions.I think i made some mistaks.


Now there are three problems i found.

1. Invalid argument

a running guest
[root@localhost ~]# virsh list
 Id    Name                           State
----------------------------------------------------
 2     test-1                         running
virsh memtune test-1
hard_limit     : 1283748
soft_limit     : unlimited
swap_hard_limit: unlimited

#virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020
error: Unable to change memory parameters
error: unable to set memory swap_hard_limit tunable: Invalid argument

#virsh memtune test-1
hard_limit     : 1283748
soft_limit     : 100000
swap_hard_limit: unlimited

[root@localhost ~]# virsh list 
 Id    Name                           State
----------------------------------------------------
 2     test-1                         running

libvirtd.log:2012-09-03 10:16:56.136+0000: 2501: error : qemuDomainSetMemoryParameters:7229 : unable to set memory hard_limit tunable: Device or resource busy
2012-09-03 10:16:56.136+0000: 2501: error : qemuDomainSetMemoryParameters:7255 : unable to set swap_hard_limit tunable: Invalid argument

/*And just repeat excute the command again , it will succus*/

#virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020
[root@localhost ~]# virsh memtune test-1
hard_limit     : 1000000
soft_limit     : 1000000
swap_hard_limit: 1000020

[root@localhost ~]# virsh list 
 Id    Name                           State
----------------------------------------------------
 2     test-1                         running


2.Connection reset by peer

Tips:I only can reproduce it on local  with NOT 100% .If use ssh to connect a remote  one , the error won't happen , at least i don't found yet.

#virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020

# virsh memtune test-1
hard_limit     : 100000
soft_limit     : 100000
swap_hard_limit: 100020

# virsh list
 Id    Name                           State
----------------------------------------------------
 6     test-1                         running

# virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100090 --live

# virsh memtune test-1
hard_limit     : 100000
soft_limit     : 100000
swap_hard_limit: 100020

# virsh list
 Id    Name                           State
----------------------------------------------------


libvirtd.log:2012-09-03 10:30:30.529+0000: 2498: error : qemuMonitorIORead:511 : Unable to read from monitor: Connection reset by peer


3.Memory cgroup out of memory
Till now , i have no idea how and when the situation will happen.Just sometimes i destroy a running guest after do some memtune actions ,then the error occured.

I catch a kernel log, see attachenment .
Comment 14 Luwen Su 2012-09-03 06:52:06 EDT
Created attachment 609320 [details]
kernel-log
Comment 15 Osier Yang 2012-09-04 04:35:48 EDT
(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #11)
> I'm sorry about my comments make you confuse.You're right , it's fine for a
> shutdown guest that do any memtune actions.I think i made some mistaks.
> 
> 
> Now there are three problems i found.
> 
> 1. Invalid argument
> 
> a running guest
> [root@localhost ~]# virsh list
>  Id    Name                           State
> ----------------------------------------------------
>  2     test-1                         running
> virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : unlimited
> swap_hard_limit: unlimited
> 
> #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100020
> error: Unable to change memory parameters
> error: unable to set memory swap_hard_limit tunable: Invalid argument
> 
> #virsh memtune test-1
> hard_limit     : 1283748
> soft_limit     : 100000
> swap_hard_limit: unlimited
> 
> [root@localhost ~]# virsh list 
>  Id    Name                           State
> ----------------------------------------------------
>  2     test-1                         running
> 
> libvirtd.log:2012-09-03 10:16:56.136+0000: 2501: error :
> qemuDomainSetMemoryParameters:7229 : unable to set memory hard_limit
> tunable: Device or resource busy
> 2012-09-03 10:16:56.136+0000: 2501: error :
> qemuDomainSetMemoryParameters:7255 : unable to set swap_hard_limit tunable:
> Invalid argument
> 
> /*And just repeat excute the command again , it will succus*/
> 
> #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100020
> [root@localhost ~]# virsh memtune test-1
> hard_limit     : 1000000
> soft_limit     : 1000000
> swap_hard_limit: 1000020

Interesting, why the limits is equal to what you specified multiplied by 10?

> 
> [root@localhost ~]# virsh list 
>  Id    Name                           State
> ----------------------------------------------------
>  2     test-1                         running
> 
> 
> 2.Connection reset by peer
> 
> Tips:I only can reproduce it on local  with NOT 100% .If use ssh to connect
> a remote  one , the error won't happen , at least i don't found yet.
> 
> #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100020
> 
> # virsh memtune test-1
> hard_limit     : 100000
> soft_limit     : 100000
> swap_hard_limit: 100020
> 
> # virsh list
>  Id    Name                           State
> ----------------------------------------------------
>  6     test-1                         running
> 
> # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> --swap-hard-limit 100090 --live
> 
> # virsh memtune test-1
> hard_limit     : 100000
> soft_limit     : 100000
> swap_hard_limit: 100020
> 
> # virsh list
>  Id    Name                           State
> ----------------------------------------------------
> 
> 
> libvirtd.log:2012-09-03 10:30:30.529+0000: 2498: error :
> qemuMonitorIORead:511 : Unable to read from monitor: Connection reset by peer
> 

I'm not sure if the qemu crashed becaused the low memory limits (not a bug if yes), and anyway, regardless of whether it's a bug or not, it's not related
with cgroup setting.
 
> 
> 3.Memory cgroup out of memory

Please use the real error message.

> Till now , i have no idea how and when the situation will happen.Just
> sometimes i destroy a running guest after do some memtune actions ,then the
> error occured.

The reason for out of memory can be plenty many. And surely it's not related
with memtune, unless we have crazy codes which eat up the memory. 

> 
> I catch a kernel log, see attachenment .
Comment 16 Osier Yang 2012-09-04 04:49:48 EDT
(In reply to comment #15)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > (In reply to comment #11)
> > I'm sorry about my comments make you confuse.You're right , it's fine for a
> > shutdown guest that do any memtune actions.I think i made some mistaks.
> > 
> > 
> > Now there are three problems i found.
> > 
> > 1. Invalid argument
> > 
> > a running guest
> > [root@localhost ~]# virsh list
> >  Id    Name                           State
> > ----------------------------------------------------
> >  2     test-1                         running
> > virsh memtune test-1
> > hard_limit     : 1283748
> > soft_limit     : unlimited
> > swap_hard_limit: unlimited
> > 
> > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> > --swap-hard-limit 100020
> > error: Unable to change memory parameters
> > error: unable to set memory swap_hard_limit tunable: Invalid argument

I'm quite sure the error is from cgroup.

> > 
> > #virsh memtune test-1
> > hard_limit     : 1283748
> > soft_limit     : 100000
> > swap_hard_limit: unlimited
> > 
> > [root@localhost ~]# virsh list 
> >  Id    Name                           State
> > ----------------------------------------------------
> >  2     test-1                         running
> > 
> > libvirtd.log:2012-09-03 10:16:56.136+0000: 2501: error :
> > qemuDomainSetMemoryParameters:7229 : unable to set memory hard_limit
> > tunable: Device or resource busy
> > 2012-09-03 10:16:56.136+0000: 2501: error :
> > qemuDomainSetMemoryParameters:7255 : unable to set swap_hard_limit tunable:
> > Invalid argument
> > 
> > /*And just repeat excute the command again , it will succus*/
> > 
> > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> > --swap-hard-limit 100020
> > [root@localhost ~]# virsh memtune test-1
> > hard_limit     : 1000000
> > soft_limit     : 1000000
> > swap_hard_limit: 1000020
> 
> Interesting, why the limits is equal to what you specified multiplied by 10?
> 
> > 
> > [root@localhost ~]# virsh list 
> >  Id    Name                           State
> > ----------------------------------------------------
> >  2     test-1                         running
> > 
> > 
> > 2.Connection reset by peer
> > 
> > Tips:I only can reproduce it on local  with NOT 100% .If use ssh to connect
> > a remote  one , the error won't happen , at least i don't found yet.
> > 
> > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> > --swap-hard-limit 100020
> > 
> > # virsh memtune test-1
> > hard_limit     : 100000
> > soft_limit     : 100000
> > swap_hard_limit: 100020
> > 
> > # virsh list
> >  Id    Name                           State
> > ----------------------------------------------------
> >  6     test-1                         running
> > 
> > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000
> > --swap-hard-limit 100090 --live
> > 
> > # virsh memtune test-1
> > hard_limit     : 100000
> > soft_limit     : 100000
> > swap_hard_limit: 100020
> > 
> > # virsh list
> >  Id    Name                           State
> > ----------------------------------------------------
> > 
> > 
> > libvirtd.log:2012-09-03 10:30:30.529+0000: 2498: error :
> > qemuMonitorIORead:511 : Unable to read from monitor: Connection reset by peer
> > 
> 
> I'm not sure if the qemu crashed becaused the low memory limits (not a bug
> if yes), and anyway, regardless of whether it's a bug or not, it's not
> related
> with cgroup setting.
>  
> > 
> > 3.Memory cgroup out of memory
> 
> Please use the real error message.
> 
> > Till now , i have no idea how and when the situation will happen.Just
> > sometimes i destroy a running guest after do some memtune actions ,then the
> > error occured.
> 
> The reason for out of memory can be plenty many. And surely it's not related
> with memtune, unless we have crazy codes which eat up the memory. 
> 
> > 
> > I catch a kernel log, see attachenment .
Comment 18 Luwen Su 2012-09-04 05:31:34 EDT
According comments 13 ,15,16 , it's not memtune's issue , so verify the bug .
Comment 19 errata-xmlrpc 2013-02-21 02:19:11 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html

Note You need to log in before you can comment on or make changes to this bug.