Bug 839537
| Summary: | Error occurs when given hard_limit in memtune more than current swap_hard_limit | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | hongming <honzhang> | ||||||
| Component: | libvirt | Assignee: | Osier Yang <jyang> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 6.4 | CC: | acathrow, ajia, dallan, dyasny, dyuan, jyang, lsu, mzhan, rwu | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | libvirt-0.10.1-1.el6 | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2013-02-21 07:19:11 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 826880 | ||||||||
| Attachments: |
|
||||||||
|
Description
hongming
2012-07-12 08:57:55 UTC
Patch posted upstream: https://www.redhat.com/archives/libvir-list/2012-July/msg00535.html *** Bug 767921 has been marked as a duplicate of this bug. *** Commit 75e5bec97 fixed the problem. Seems there are some problems.So , accroding my steps , should we open a new bz or keep this one to track the problem? ------------------------------------------------------------------ first for verfy the bug: # virsh start test-1 Domain test-1 started # virsh memtune test-1 hard_limit : 1283748 soft_limit : unlimited swap_hard_limit: unlimited # virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020 --swap-hard-limit 1000020 --live # virsh memtune test-1 hard_limit : 1000020 soft_limit : 1000020 swap_hard_limit: 1000020 # virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020 --swap-hard-limit 1000040 --live # virsh memtune test-1 hard_limit : 1000032 soft_limit : 1000020 swap_hard_limit: 1000040 # virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020 --swap-hard-limit 1000040 --live error: Unable to change memory parameters error: Requested operation is not valid: domain is not running [root@localhost /]# virsh memtune test-1 hard_limit : 0 soft_limit : 0 swap_hard_limit: 0 ------------------------------------------------------------------------ Below are other situations i test. 1. a shutoff guest # virsh memtune test-1 hard_limit : 1283748 soft_limit : unlimited swap_hard_limit: unlimited # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020 error: Unable to change memory parameters error: End of file while reading data: Input/output error error: Failed to reconnect to the hypervisor # service libvirtd status libvirtd dead but pid file exists libvirtd.log:see the attachment 2 #service libvirtd restart #virsh start test-1 # virsh memtune test-1 hard_limit : 1283748 soft_limit : unlimited swap_hard_limit: unlimited # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020 the guest is atuo be destroyed libvirtd.log: 2012-08-24 04:20:03.076+0000: 6090: info : libvirt version: 0.10.0, package: 0rc1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-08-23-00:04:57, x86-007.build.bos.redhat.com) 2012-08-24 04:20:03.076+0000: 6090: error : virDBusGetSystemBus:193 : internal error DBus support not compiled into this binary 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 3 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 4 2012-08-24 04:20:09.029+0000: 6080: warning : qemuSetupCgroupForEmulator:659 : cgroup 5 is not active 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 6 2012-08-24 04:20:26.062+0000: 6079: error : qemuMonitorIO:602 : internal error End of file from monitor 3. # virsh start test-1 Domain test-1 started # virsh memtune test-1 hard_limit : 1283748 soft_limit : unlimited swap_hard_limit: unlimited # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100021 --live # virsh memtune test-1 hard_limit : 100000 soft_limit : 100000 swap_hard_limit: 100024 libvirtd.log: 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 3 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 4 2012-08-24 04:21:56.626+0000: 6080: warning : qemuSetupCgroupForEmulator:659 : cgroup 5 is not active 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm cgroup in controller 6 4. virsh start test-1 Domain test-1 started # virsh memtune test-1 hard_limit : 1283748 soft_limit : unlimited swap_hard_limit: unlimited # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100025 --live # virsh memtune test-1 hard_limit : 100000 soft_limit : 100000 swap_hard_limit: 100028 <----different with 25 # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100026 --live # virsh memtune test-1 hard_limit : 100000 soft_limit : 100000 swap_hard_limit: 100028 # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100029 --live # virsh memtune test-1 hard_limit : 100000 soft_limit : 100000 swap_hard_limit: 100032 # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100031 --live error: Unable to change memory parameters error: Requested operation is not valid: domain is not running # virsh memtune test-1 hard_limit : 0 soft_limit : 0 swap_hard_limit: 0 Created attachment 606749 [details]
memtune fail
(In reply to comment #6) > Seems there are some problems.So , accroding my steps , should we open a > new bz or keep this one to track the problem? > > ------------------------------------------------------------------ > first for verfy the bug: > # virsh start test-1 > Domain test-1 started > # virsh memtune test-1 > hard_limit : 1283748 > soft_limit : unlimited > swap_hard_limit: unlimited > > # virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020 > --swap-hard-limit 1000020 --live > > # virsh memtune test-1 > hard_limit : 1000020 > soft_limit : 1000020 > swap_hard_limit: 1000020 > > # virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020 > --swap-hard-limit 1000040 --live > > # virsh memtune test-1 > hard_limit : 1000032 > soft_limit : 1000020 > swap_hard_limit: 1000040 > > # virsh memtune test-1 --hard-limit 1000020 --soft-limit 1000020 > --swap-hard-limit 1000040 --live > error: Unable to change memory parameters > error: Requested operation is not valid: domain is not running So what's the domain state at this point? could it be shutdown by another thread? > > [root@localhost /]# virsh memtune test-1 > hard_limit : 0 > soft_limit : 0 > swap_hard_limit: 0 > > ------------------------------------------------------------------------ > Below are other situations i test. > > > 1. > a shutoff guest > # virsh memtune test-1 > hard_limit : 1283748 > soft_limit : unlimited > swap_hard_limit: unlimited > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100020 > error: Unable to change memory parameters > error: End of file while reading data: Input/output error > error: Failed to reconnect to the hypervisor Could you attach the gdb backstrace? > > # service libvirtd status > libvirtd dead but pid file exists > > libvirtd.log:see the attachment > > > 2 [details] > #service libvirtd restart > #virsh start test-1 > # virsh memtune test-1 > hard_limit : 1283748 > soft_limit : unlimited > swap_hard_limit: unlimited > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100020 > the guest is atuo be destroyed > libvirtd.log: > 2012-08-24 04:20:03.076+0000: 6090: info : libvirt version: 0.10.0, package: > 0rc1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, > 2012-08-23-00:04:57, x86-007.build.bos.redhat.com) > 2012-08-24 04:20:03.076+0000: 6090: error : virDBusGetSystemBus:193 : > internal error DBus support not compiled into this binary > 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm > cgroup in controller 3 > 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm > cgroup in controller 4 > 2012-08-24 04:20:09.029+0000: 6080: warning : qemuSetupCgroupForEmulator:659 > : cgroup 5 is not active > 2012-08-24 04:20:09.029+0000: 6080: warning : virCgroupMoveTask:885 : no vm > cgroup in controller 6 > 2012-08-24 04:20:26.062+0000: 6079: error : qemuMonitorIO:602 : internal > error End of file from monitor This tells the domain is shutdown. I'm wondering what causes that. > > > 3. > # virsh start test-1 > Domain test-1 started > # virsh memtune test-1 > hard_limit : 1283748 > soft_limit : unlimited > swap_hard_limit: unlimited > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100021 --live > > # virsh memtune test-1 > hard_limit : 100000 > soft_limit : 100000 > swap_hard_limit: 100024 > > libvirtd.log: > 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm > cgroup in controller 3 > 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm > cgroup in controller 4 > 2012-08-24 04:21:56.626+0000: 6080: warning : qemuSetupCgroupForEmulator:659 > : cgroup 5 is not active > 2012-08-24 04:21:56.626+0000: 6080: warning : virCgroupMoveTask:885 : no vm > cgroup in controller 6 > > 4. > virsh start test-1 > Domain test-1 started > > # virsh memtune test-1 > hard_limit : 1283748 > soft_limit : unlimited > swap_hard_limit: unlimited > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100025 --live > > # virsh memtune test-1 > hard_limit : 100000 > soft_limit : 100000 > swap_hard_limit: 100028 <----different with 25 Fine, should be rounded up by cgroup. > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100026 --live > > # virsh memtune test-1 > hard_limit : 100000 > soft_limit : 100000 > swap_hard_limit: 100028 > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100029 --live > > # virsh memtune test-1 > hard_limit : 100000 > soft_limit : 100000 > swap_hard_limit: 100032 > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100031 --live > error: Unable to change memory parameters > error: Requested operation is not valid: domain is not running > > # virsh memtune test-1 > hard_limit : 0 > soft_limit : 0 > swap_hard_limit: 0 (In reply to comment #8) > (In reply to comment #6) > > Seems there are some problems.So , accroding my steps , should we open a > > new bz or keep this one to track the problem? I test the the situations above again 1.All the behavior cause libvirtd crashed have the same gdb information: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ff06b65c700 (LWP 19506)] 0x0000003de484a257 in virCgroupPathOfController () from /usr/lib64/libvirt.so.0 (gdb) backtrace #0 0x0000003de484a257 in virCgroupPathOfController () from /usr/lib64/libvirt.so.0 #1 0x0000003de484a733 in ?? () from /usr/lib64/libvirt.so.0 #2 0x0000003de484b12c in ?? () from /usr/lib64/libvirt.so.0 #3 0x0000003de484b1ee in virCgroupGetMemSwapHardLimit () from /usr/lib64/libvirt.so.0 #4 0x00000000004646bd in ?? () #5 0x0000003de48f6640 in virDomainSetMemoryParameters () from /usr/lib64/libvirt.so.0 #6 0x000000000043f079 in ?? () #7 0x0000003de492ee3d in virNetServerProgramDispatch () from /usr/lib64/libvirt.so.0 #8 0x0000003de492fade in ?? () from /usr/lib64/libvirt.so.0 #9 0x0000003de493017c in ?? () from /usr/lib64/libvirt.so.0 #10 0x0000003de485f30c in ?? () from /usr/lib64/libvirt.so.0 #11 0x0000003de485ebf9 in ?? () from /usr/lib64/libvirt.so.0 #12 0x0000003dcd407851 in start_thread () from /lib64/libpthread.so.0 #13 0x0000003dcd0e767d in clone () from /lib64/libc.so.6 2.For the Verify steps , it works fine and i can't reproduce it now . I still research what the reason cause failure. commit 20b780aec9d48888c5088b97887d268df77884c0
Author: Osier Yang <jyang>
Date: Wed Aug 29 14:46:06 2012 +0800
qemu: Sort the numa params only when it affects the live config
As the next boot doesn't have to worry about the previous numa
params setting (there is no).
A follow up patch to fix the crash.
Test with libvirt-0.10.1.1 again. Good news is libvirt don't cras. But it seems still have some issues.So change to assign. libvirt-0.10.1-1.el6.x86_64 qemu-kvm-rhev-0.12.1.2-2.307.el6.x86_64 kernel-2.6.32-298.el6.x86_64 1. virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020 --swap-hard-limit 1000040 --live # virsh memtune test-1 hard_limit : 1000032 <----the number doesn't accurate yet soft_limit : 1000020 swap_hard_limit: 1000040 the libvirtd.log's warning still exist warning : virCgroupMoveTask:885 : no vm cgroup in controller 3 2. a shutoff guest # virsh memtune test-1 hard_limit : 1283748 soft_limit : unlimited swap_hard_limit: unlimited # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020 It works , but qemu hang this time then libvirt get a Timed out during operation: cannot acquire state change lock libvirtd.log (omit the time info): warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process 18194, sending SIGKILL warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process 18194 info : libvirt version: 0.10.1, package: 1.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-08-31-10:30:29, x86-009.build.bos.redhat.com) error : virNetSocketReadWire:1184 : End of file while reading data: Input/output error warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process 18194, sending SIGKILL warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process 18194 warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process 18194, sending SIGKILL warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process 18194 warning : qemuDomainObjBeginJobInternal:838 : Cannot start job (destroy, none) for domain test; current job is (modify, none) owned by (18474, 0) error : qemuDomainObjBeginJobInternal:842 : Timed out during operation: cannot acquire state change lock (In reply to comment #11) > Test with libvirt-0.10.1.1 again. > Good news is libvirt don't cras. > But it seems still have some issues.So change to assign. > > libvirt-0.10.1-1.el6.x86_64 > qemu-kvm-rhev-0.12.1.2-2.307.el6.x86_64 > kernel-2.6.32-298.el6.x86_64 > > 1. > virsh memtune test-1 --hard-limit 1000030 --soft-limit 1000020 > --swap-hard-limit 1000040 --live > > # virsh memtune test-1 > hard_limit : 1000032 <----the number doesn't accurate yet See my comment in comment 8, you will known that if you tested with cgroup directly. > soft_limit : 1000020 > swap_hard_limit: 1000040 > > the libvirtd.log's warning still exist > warning : virCgroupMoveTask:885 : no vm cgroup in controller 3 A warning can't be a bug, unless there is something crazy there. And the warning is not caused by setting cgroup for emulartor, it's not from memory tuning cgroup. So if you think the warning has some problems, file a separate bug. > > 2. > a shutoff guest > # virsh memtune test-1 > hard_limit : 1283748 > soft_limit : unlimited > swap_hard_limit: unlimited > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100020 > > It works , but qemu hang this time then libvirt get a > Timed out during operation: cannot acquire state change lock I don't understand it here. Mainly questions are: 1) what does "it works" mean here, does it mean 'memtune' return successfully? or it hangs there? 2) what you did right before and after "memtune". As the error "timeout to acquire the lock" is about killing the domain process. I don't think it occurs when doing "memtune", as you declared the domain was already in shutoff state before executing "memtune" command. 3) I don't see any dead lock possbility in "memtune" API. It's very likely you did something else, that's why what I asked "what you did right before and after memtune command". > > libvirtd.log (omit the time info): > warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process > 18194, sending SIGKILL > warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process > 18194 > info : libvirt version: 0.10.1, package: 1.el6 (Red Hat, Inc. > <http://bugzilla.redhat.com/bugzilla>, 2012-08-31-10:30:29, > x86-009.build.bos.redhat.com) > > error : virNetSocketReadWire:1184 : End of file while reading data: > Input/output error > warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process > 18194, sending SIGKILL > warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process > 18194 > warning : qemuProcessKill:3966 : Timed out waiting after SIGTERM to process > 18194, sending SIGKILL > warning : qemuProcessKill:3998 : Timed out waiting after SIGKILL to process > 18194 > warning : qemuDomainObjBeginJobInternal:838 : Cannot start job (destroy, > none) for domain test; current job is (modify, none) owned by (18474, 0) > error : qemuDomainObjBeginJobInternal:842 : Timed out during operation: > cannot acquire state change lock (In reply to comment #12) > (In reply to comment #11) I'm sorry about my comments make you confuse.You're right , it's fine for a shutdown guest that do any memtune actions.I think i made some mistaks. Now there are three problems i found. 1. Invalid argument a running guest [root@localhost ~]# virsh list Id Name State ---------------------------------------------------- 2 test-1 running virsh memtune test-1 hard_limit : 1283748 soft_limit : unlimited swap_hard_limit: unlimited #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020 error: Unable to change memory parameters error: unable to set memory swap_hard_limit tunable: Invalid argument #virsh memtune test-1 hard_limit : 1283748 soft_limit : 100000 swap_hard_limit: unlimited [root@localhost ~]# virsh list Id Name State ---------------------------------------------------- 2 test-1 running libvirtd.log:2012-09-03 10:16:56.136+0000: 2501: error : qemuDomainSetMemoryParameters:7229 : unable to set memory hard_limit tunable: Device or resource busy 2012-09-03 10:16:56.136+0000: 2501: error : qemuDomainSetMemoryParameters:7255 : unable to set swap_hard_limit tunable: Invalid argument /*And just repeat excute the command again , it will succus*/ #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020 [root@localhost ~]# virsh memtune test-1 hard_limit : 1000000 soft_limit : 1000000 swap_hard_limit: 1000020 [root@localhost ~]# virsh list Id Name State ---------------------------------------------------- 2 test-1 running 2.Connection reset by peer Tips:I only can reproduce it on local with NOT 100% .If use ssh to connect a remote one , the error won't happen , at least i don't found yet. #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100020 # virsh memtune test-1 hard_limit : 100000 soft_limit : 100000 swap_hard_limit: 100020 # virsh list Id Name State ---------------------------------------------------- 6 test-1 running # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 --swap-hard-limit 100090 --live # virsh memtune test-1 hard_limit : 100000 soft_limit : 100000 swap_hard_limit: 100020 # virsh list Id Name State ---------------------------------------------------- libvirtd.log:2012-09-03 10:30:30.529+0000: 2498: error : qemuMonitorIORead:511 : Unable to read from monitor: Connection reset by peer 3.Memory cgroup out of memory Till now , i have no idea how and when the situation will happen.Just sometimes i destroy a running guest after do some memtune actions ,then the error occured. I catch a kernel log, see attachenment . Created attachment 609320 [details]
kernel-log
(In reply to comment #13) > (In reply to comment #12) > > (In reply to comment #11) > I'm sorry about my comments make you confuse.You're right , it's fine for a > shutdown guest that do any memtune actions.I think i made some mistaks. > > > Now there are three problems i found. > > 1. Invalid argument > > a running guest > [root@localhost ~]# virsh list > Id Name State > ---------------------------------------------------- > 2 test-1 running > virsh memtune test-1 > hard_limit : 1283748 > soft_limit : unlimited > swap_hard_limit: unlimited > > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100020 > error: Unable to change memory parameters > error: unable to set memory swap_hard_limit tunable: Invalid argument > > #virsh memtune test-1 > hard_limit : 1283748 > soft_limit : 100000 > swap_hard_limit: unlimited > > [root@localhost ~]# virsh list > Id Name State > ---------------------------------------------------- > 2 test-1 running > > libvirtd.log:2012-09-03 10:16:56.136+0000: 2501: error : > qemuDomainSetMemoryParameters:7229 : unable to set memory hard_limit > tunable: Device or resource busy > 2012-09-03 10:16:56.136+0000: 2501: error : > qemuDomainSetMemoryParameters:7255 : unable to set swap_hard_limit tunable: > Invalid argument > > /*And just repeat excute the command again , it will succus*/ > > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100020 > [root@localhost ~]# virsh memtune test-1 > hard_limit : 1000000 > soft_limit : 1000000 > swap_hard_limit: 1000020 Interesting, why the limits is equal to what you specified multiplied by 10? > > [root@localhost ~]# virsh list > Id Name State > ---------------------------------------------------- > 2 test-1 running > > > 2.Connection reset by peer > > Tips:I only can reproduce it on local with NOT 100% .If use ssh to connect > a remote one , the error won't happen , at least i don't found yet. > > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100020 > > # virsh memtune test-1 > hard_limit : 100000 > soft_limit : 100000 > swap_hard_limit: 100020 > > # virsh list > Id Name State > ---------------------------------------------------- > 6 test-1 running > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > --swap-hard-limit 100090 --live > > # virsh memtune test-1 > hard_limit : 100000 > soft_limit : 100000 > swap_hard_limit: 100020 > > # virsh list > Id Name State > ---------------------------------------------------- > > > libvirtd.log:2012-09-03 10:30:30.529+0000: 2498: error : > qemuMonitorIORead:511 : Unable to read from monitor: Connection reset by peer > I'm not sure if the qemu crashed becaused the low memory limits (not a bug if yes), and anyway, regardless of whether it's a bug or not, it's not related with cgroup setting. > > 3.Memory cgroup out of memory Please use the real error message. > Till now , i have no idea how and when the situation will happen.Just > sometimes i destroy a running guest after do some memtune actions ,then the > error occured. The reason for out of memory can be plenty many. And surely it's not related with memtune, unless we have crazy codes which eat up the memory. > > I catch a kernel log, see attachenment . (In reply to comment #15) > (In reply to comment #13) > > (In reply to comment #12) > > > (In reply to comment #11) > > I'm sorry about my comments make you confuse.You're right , it's fine for a > > shutdown guest that do any memtune actions.I think i made some mistaks. > > > > > > Now there are three problems i found. > > > > 1. Invalid argument > > > > a running guest > > [root@localhost ~]# virsh list > > Id Name State > > ---------------------------------------------------- > > 2 test-1 running > > virsh memtune test-1 > > hard_limit : 1283748 > > soft_limit : unlimited > > swap_hard_limit: unlimited > > > > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > > --swap-hard-limit 100020 > > error: Unable to change memory parameters > > error: unable to set memory swap_hard_limit tunable: Invalid argument I'm quite sure the error is from cgroup. > > > > #virsh memtune test-1 > > hard_limit : 1283748 > > soft_limit : 100000 > > swap_hard_limit: unlimited > > > > [root@localhost ~]# virsh list > > Id Name State > > ---------------------------------------------------- > > 2 test-1 running > > > > libvirtd.log:2012-09-03 10:16:56.136+0000: 2501: error : > > qemuDomainSetMemoryParameters:7229 : unable to set memory hard_limit > > tunable: Device or resource busy > > 2012-09-03 10:16:56.136+0000: 2501: error : > > qemuDomainSetMemoryParameters:7255 : unable to set swap_hard_limit tunable: > > Invalid argument > > > > /*And just repeat excute the command again , it will succus*/ > > > > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > > --swap-hard-limit 100020 > > [root@localhost ~]# virsh memtune test-1 > > hard_limit : 1000000 > > soft_limit : 1000000 > > swap_hard_limit: 1000020 > > Interesting, why the limits is equal to what you specified multiplied by 10? > > > > > [root@localhost ~]# virsh list > > Id Name State > > ---------------------------------------------------- > > 2 test-1 running > > > > > > 2.Connection reset by peer > > > > Tips:I only can reproduce it on local with NOT 100% .If use ssh to connect > > a remote one , the error won't happen , at least i don't found yet. > > > > #virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > > --swap-hard-limit 100020 > > > > # virsh memtune test-1 > > hard_limit : 100000 > > soft_limit : 100000 > > swap_hard_limit: 100020 > > > > # virsh list > > Id Name State > > ---------------------------------------------------- > > 6 test-1 running > > > > # virsh memtune test-1 --hard-limit 100000 --soft-limit 100000 > > --swap-hard-limit 100090 --live > > > > # virsh memtune test-1 > > hard_limit : 100000 > > soft_limit : 100000 > > swap_hard_limit: 100020 > > > > # virsh list > > Id Name State > > ---------------------------------------------------- > > > > > > libvirtd.log:2012-09-03 10:30:30.529+0000: 2498: error : > > qemuMonitorIORead:511 : Unable to read from monitor: Connection reset by peer > > > > I'm not sure if the qemu crashed becaused the low memory limits (not a bug > if yes), and anyway, regardless of whether it's a bug or not, it's not > related > with cgroup setting. > > > > > 3.Memory cgroup out of memory > > Please use the real error message. > > > Till now , i have no idea how and when the situation will happen.Just > > sometimes i destroy a running guest after do some memtune actions ,then the > > error occured. > > The reason for out of memory can be plenty many. And surely it's not related > with memtune, unless we have crazy codes which eat up the memory. > > > > > I catch a kernel log, see attachenment . Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0276.html |