Bug 1168866
| Summary: | "libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy" | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jaroslav Suchanek <jsuchane> |
| Component: | libvirt | Assignee: | Martin Kletzander <mkletzan> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.1 | CC: | agedosier, berrange, clalancette, dkelson, dyuan, extras-qa, honzhang, itamar, jdenemar, jforbes, jmiao, kchamart, laine, libvirt-maint, mkletzan, mzhan, ndipanov, rbalakri, shyu, veillard, virt-maint |
| Target Milestone: | rc | Keywords: | Upstream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-1.2.8-10.el7 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1168672 | Environment: | |
| Last Closed: | 2015-03-05 07:47:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1168672, 1168944 | ||
| Bug Blocks: | |||
|
Description
Jaroslav Suchanek
2014-11-28 09:41:19 UTC
Fixed upstream with v1.2.10-75-gc6e9024:
commit c6e90248676126c209b3b6017ad27cf6c6a0ab8f
Author: Wang Rui <moon.wangrui>
Date: Mon Nov 10 21:53:19 2014 +0800
qemu: fix domain startup failing with 'strict' mode in numatune
*** Bug 1168944 has been marked as a duplicate of this bug. *** *** Bug 1118517 has been marked as a duplicate of this bug. *** Hi Kashyap， I am libvirt qe, and trying to reproduce this bug on RHEL7. Could you meet this problem on RHEL7? Is nested KVM necessary for this bug? (In reply to Jincheng Miao from comment #6) > Hi Kashyap， Hi Jincheng, > > I am libvirt qe, and trying to reproduce this bug on RHEL7. > Could you meet this problem on RHEL7? I didn't test it on RHEL7, but it should be reproducible without this fix on RHEL7, too, when you try to boot a libvirt guest with a single NUMA cell requested. I hit this bug while booting libvirt guest in the context of OpenStack in a developer environment (DevStack) on Fedora-21, you can see how I tested it here (it's a bit involved procedure): https://bugzilla.redhat.com/show_bug.cgi?id=1168672#c8 > Is nested KVM necessary for this bug? Nested KVM is not mandatory (but my test environment involved nested KVM). However, you do need a physical machine with more than a single NUMA cell. You can find that out by running: $ virsh nodeinfo You can probably try to reproduce this issue by trying to boot a libvirt guest with a single NUMA node. You can see the QEMU CLI from the parent bug, here: https://bugzilla.redhat.com/show_bug.cgi?id=1168672#c9 Specifically, notice this from the QEMU CLI: [. . .] -object memory-backend-ram,size=1024M,id=ram-node0,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 [. . .] Hi Martin and Kashyap,
I tested it on a machine with 2 nodes, and configured 1 guest node.
...
<vcpu placement='auto'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0-3'/>
<vcpupin vcpu='1' cpuset='0-3'/>
<vcpupin vcpu='2' cpuset='0-3'/>
<vcpupin vcpu='3' cpuset='0-3'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
...
<cpu mode='host-passthrough'>
<numa>
<cell id='0' cpus='0-3' memory='1048576'/>
</numa>
</cpu>
...
But I got
"
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2db.scope/vcpu0/cpuset.cpus': Permission denied
"
It is not 'cpuset.mems: Device or resource busy'.
Does it same with this bug?
No, permission denied is something completely different. I don't even know how this is possible :) (In reply to Martin Kletzander from comment #9) > No, permission denied is something completely different. I don't even know > how this is possible :) OK, I already filed a bug 1174125 for "cpuset.cpus: Permission denied" problem. This bug is cloned from fedora, so it couldn't be reproduced on RHEL.
But I have one method to reproduce it, using package libvirt-1.2.8-10.el7.
Rebuild libvirt-1.2.8-10.el7 with 411cea6 and without c6e9024
commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
Author: Zhou yimin <zhouyimin>
Date: Thu Oct 16 22:18:48 2014 +0800
qemu: move setting emulatorpin ahead of monitor showing up
commit c6e90248676126c209b3b6017ad27cf6c6a0ab8f
Author: Wang Rui <moon.wangrui>
Date: Mon Nov 10 21:53:19 2014 +0800
qemu: fix domain startup failing with 'strict' mode in numatune
After that, I could reproduce this problem:
# virsh edit b
...
<vcpu placement='auto'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1-3'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
</numatune>
...
<cpu mode='host-passthrough'>
<numa>
<cell id='0' cpus='0-3' memory='1048576'/>
</numa>
</cpu>
...
# virsh start b
error: Failed to start domain b
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2db.scope/cpuset.mems': Device or resource busy
As commit c6e9024 said:
"
It's broken by Commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
which moved qemuSetupCgroupForEmulator() before setting cpuset.mems
in qemuSetupCgroupPostInit.
Directory '$cgroup_path/emulator/' is created in qemuSetupCgroupForEmulator.
But '$cgroup_path/emulator/cpuset.mems' it not set and has a default value
(all nodes, such as 0-1). Then we setup '$cgroup_path/cpuset.mems' to the
nodemask (in this case it's '0') in qemuSetupCgroupPostInit. It must fail.
"
The problem "cpuset.cpus: Permission denied" is due to the inconsistent between numad and vcpupin user specified. I will post log in bug 1174125. So this commit from upstream should be add:
commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
Author: Zhou yimin <zhouyimin>
Date: Thu Oct 16 22:18:48 2014 +0800
qemu: move setting emulatorpin ahead of monitor showing up
Yeah, it will be added for bug 1170484. Since this upstream commit 411cea6 is backported in latest libvirt-1.2.8-11.el7.x86_64, I will use it to test this bug.
# rpm -q libvirt
libvirt-1.2.8-11.el7.x86_64
1. using the same guest configuration:
# virsh edit b
...
<vcpu placement='static'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1-3'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
</numatune>
...
<cpu mode='host-passthrough'>
<numa>
<cell id='0' cpus='0-3' memory='1048576'/>
</numa>
</cpu>
...
2. start guest
# virsh start b
Domain b started
Using vcpu placement='auto' also triggers "cpuset.cpus: Permission denied", but bug 1174125 will cover it.
So I will change the status to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html |